date:20160413

[jira] [Created] (KYLIN-1584) Specify region cut size in cubedesc and leave the RealizationCapacity in model as a hint

2016-04-13 Thread hongbin ma (JIRA)

hongbin ma created KYLIN-1584:
-

 Summary: Specify region cut size in cubedesc and leave the 
RealizationCapacity in model as a hint
 Key: KYLIN-1584
 URL: https://issues.apache.org/jira/browse/KYLIN-1584
 Project: Kylin
  Issue Type: Improvement
Reporter: hongbin ma
Assignee: hongbin ma


currently the cube region's size is determined in model's 
capacity(RealizationCapacity), it will enforce different realizations under the 
same model having uniformed region size. This is not wanted feature because 
cube builder might need to try out different region size to tune performance. 
I'm suggesting to add a field called "region_size" (float type, 1.0 means 1G 
per region) in cube desc to explicitly indicate the cube's region size. The 
field "region_size"  will not be included in cubedesc's signature calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: How to create a streaming model.

2016-04-13 Thread ShaoFeng Shi

It's hard to describe all steps in few words.. we owe a tutorial for this,
let me open a task for this, please watch it for any update:

https://issues.apache.org/jira/browse/KYLIN-1582

2016-04-13 19:39 GMT+08:00 陈佛林 :

> There is no example or guide to tell me how to create a streaming model.
>
> I don't know how to configure kafka topics,is it just a simple string(such
> as "topic1")?
> and where to configure  kafka address
>



-- 
Best regards,

Shaofeng Shi

Re: No detail message when Error.

2016-04-13 Thread 陈佛林

1.5.0

2016-04-13 19:41 GMT+08:00 hongbin ma :

> what is the kylin version?
>
> On Wed, Apr 13, 2016 at 7:02 PM, 陈佛林  wrote:
>
> > when i delete a model, it got "Failed to take action", got a "500"
> code.but
> > the model has been deleted at backend.
> >
> >
> > all logs when i delete a model:
> >
> > 2016-04-13 19:00:00,449 INFO  [http-bio-7070-exec-7]
> > invertedindex.IIDescManager: Initializing IIDescManager with config
> > kylin_metadata@hbase
> > 2016-04-13 19:00:00,449 INFO  [http-bio-7070-exec-7]
> > invertedindex.IIDescManager: Reloading all II desc from folder
> > kylin_metadata(key='/invertedindex_desc')@kylin_metadata@hbase
> > 2016-04-13 19:00:00,456 INFO  [http-bio-7070-exec-7]
> > metadata.MetadataManager: Dropping model 'dd'
> > 2016-04-13 19:00:00,463 INFO  [pool-8-thread-1] restclient.Broadcaster:
> new
> > broadcast event:BroadcastEvent{type=project, name=myproject,
> action=update}
> > 2016-04-13 19:00:00,483 INFO  [pool-8-thread-1] restclient.Broadcaster:
> new
> > broadcast event:BroadcastEvent{type=data_model, name=dd, action=drop}
> > 2016-04-13 19:00:00,487 INFO  [http-bio-7070-exec-8]
> > controller.CacheController: wipe cache type: PROJECT event:UPDATE
> > name:myproject
> > 2016-04-13 19:00:00,487 INFO  [http-bio-7070-exec-8]
> service.CacheService:
> > rebuild cache type: PROJECT name:myproject
> > 2016-04-13 19:00:00,490 INFO  [http-bio-7070-exec-8]
> service.CacheService:
> > removeOLAPDataSource is called for project myproject
> > 2016-04-13 19:00:00,495 INFO  [http-bio-7070-exec-8]
> > controller.CacheController: wipe cache type: DATA_MODEL event:DROP
> name:dd
> >
> > 2016-04-13 18:26 GMT+08:00 hongbin ma :
> >
> > > have you checked $KYLIN_HOME/logs/kylin.log?
> > >
> > > On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林  wrote:
> > >
> > > > hi,
> > > >
> > > > almost all error just show a little message.
> > > >
> > > > such as:
> > > >
> > > > Error Message
> > > >
> > > > Failed to take action.
> > > >
> > > >
> > > > I have no idea what happened.
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: No detail message when Error.

2016-04-13 Thread 陈佛林

when i delete a model, it got "Failed to take action", got a "500" code.but
the model has been deleted at backend.


all logs when i delete a model:

2016-04-13 19:00:00,449 INFO  [http-bio-7070-exec-7]
invertedindex.IIDescManager: Initializing IIDescManager with config
kylin_metadata@hbase
2016-04-13 19:00:00,449 INFO  [http-bio-7070-exec-7]
invertedindex.IIDescManager: Reloading all II desc from folder
kylin_metadata(key='/invertedindex_desc')@kylin_metadata@hbase
2016-04-13 19:00:00,456 INFO  [http-bio-7070-exec-7]
metadata.MetadataManager: Dropping model 'dd'
2016-04-13 19:00:00,463 INFO  [pool-8-thread-1] restclient.Broadcaster: new
broadcast event:BroadcastEvent{type=project, name=myproject, action=update}
2016-04-13 19:00:00,483 INFO  [pool-8-thread-1] restclient.Broadcaster: new
broadcast event:BroadcastEvent{type=data_model, name=dd, action=drop}
2016-04-13 19:00:00,487 INFO  [http-bio-7070-exec-8]
controller.CacheController: wipe cache type: PROJECT event:UPDATE
name:myproject
2016-04-13 19:00:00,487 INFO  [http-bio-7070-exec-8] service.CacheService:
rebuild cache type: PROJECT name:myproject
2016-04-13 19:00:00,490 INFO  [http-bio-7070-exec-8] service.CacheService:
removeOLAPDataSource is called for project myproject
2016-04-13 19:00:00,495 INFO  [http-bio-7070-exec-8]
controller.CacheController: wipe cache type: DATA_MODEL event:DROP name:dd

2016-04-13 18:26 GMT+08:00 hongbin ma :

> have you checked $KYLIN_HOME/logs/kylin.log?
>
> On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林  wrote:
>
> > hi,
> >
> > almost all error just show a little message.
> >
> > such as:
> >
> > Error Message
> >
> > Failed to take action.
> >
> >
> > I have no idea what happened.
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: ClassNotFoundException

2016-04-13 Thread hongbin ma

are you using CHD, kylin 1.5.0?

take a look at
http://apache-kylin.74782.x6.nabble.com/Stuck-with-ClassNotFoundException-HCatInputFormat-td4032.html

it can be found if you search google

On Wed, Apr 13, 2016 at 6:01 PM, 耳东 <775620...@qq.com> wrote:

> Hi all：   when I built the cube, this exception occured. Is there any
> solution to this excepiton. I am using Kylin on a cluster.
>
>
>
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class  org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1776)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
> at  org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
> ... 8  more




-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: No detail message when Error.

2016-04-13 Thread hongbin ma

have you checked $KYLIN_HOME/logs/kylin.log?

On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林  wrote:

> hi,
>
> almost all error just show a little message.
>
> such as:
>
> Error Message
>
> Failed to take action.
>
>
> I have no idea what happened.
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

No detail message when Error.

2016-04-13 Thread 陈佛林

hi,

almost all error just show a little message.

such as:

Error Message

Failed to take action.


I have no idea what happened.

ClassNotFoundException

2016-04-13 Thread ????

Hi all??   when I built the cube, this exception occured. Is there any 
solution to this excepiton. I am using Kylin on a cluster.




Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class  
org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at  
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) at  
org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at  
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at  
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at  
java.security.AccessController.doPrivileged(Native Method) at  
javax.security.auth.Subject.doAs(Subject.java:415) at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1776)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:  
java.lang.ClassNotFoundException: Class  
org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at  
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) at 
 org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) ... 8  
more

Re: build cube error

2016-04-13 Thread 陈佛林

Thanks.

When i replace the hive view with other tables, every thing is ok.

2016-04-13 17:02 GMT+08:00 wangxianbin1...@gmail.com <
wangxianbin1...@gmail.com>:

> hi!
>
> what do you mean? are you using hive view as lookup table?
>
> if it is so, it is a known issue, check this out,
> https://issues.apache.org/jira/browse/KYLIN-1077
>
> best regards!
>
>
>
> wangxianbin1...@gmail.com
>
> From: 陈佛林
> Date: 2016-04-13 16:51
> To: dev
> Subject: build cube error
> all job at yarn are SUCCESS.
>
>
> 2016-04-13 15:54:58,765 WARN  [http-bio-7070-exec-1]
> service.CacheService:108 : skip cleaning cache for
> 668468ba-f470-4c37-9b12-d66aa44cef30
>
> 2016-04-13 15:54:58,903 ERROR [pool-5-thread-6]
> execution.AbstractExecutable:62 : error execute
> HadoopShellExecutable{id=4e9d2b2a-4f20-4b35-9dc6-bf2185f43486-02,
> name=Build Dimension Dictionary, state=RUNNING}
>
> java.io.IOException: java.lang.NullPointerException
>
> at
> org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
>
> at
> org.apache.kylin.dict.lookup.SnapshotTable.(SnapshotTable.java:64)
>
> at
>
> org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
>
> at
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
>
> at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
>
> at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
>
> at
>
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>
> at
>
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
>
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
>
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
>
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
>
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.NullPointerException
>
> at
> org.apache.kylin.engine.mr.HadoopUtil.fixWindowsPath(HadoopUtil.java:78)
>
> at
> org.apache.kylin.engine.mr.HadoopUtil.makeURI(HadoopUtil.java:70)
>
> at
> org.apache.kylin.engine.mr.HadoopUtil.getFileSystem(HadoopUtil.java:65)
>
> at
>
> org.apache.kylin.engine.mr.DFSFileTable.getSizeAndLastModified(DFSFileTable.java:78)
>
> at
> org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:56)
>
> ... 16 more
>

Re: build cube error

2016-04-13 Thread wangxianbin1...@gmail.com

hi!

what do you mean? are you using hive view as lookup table?

if it is so, it is a known issue, check this out, 
https://issues.apache.org/jira/browse/KYLIN-1077 

best regards!



wangxianbin1...@gmail.com
 
From: 陈佛林
Date: 2016-04-13 16:51
To: dev
Subject: build cube error
all job at yarn are SUCCESS.
 
 
2016-04-13 15:54:58,765 WARN  [http-bio-7070-exec-1]
service.CacheService:108 : skip cleaning cache for
668468ba-f470-4c37-9b12-d66aa44cef30
 
2016-04-13 15:54:58,903 ERROR [pool-5-thread-6]
execution.AbstractExecutable:62 : error execute
HadoopShellExecutable{id=4e9d2b2a-4f20-4b35-9dc6-bf2185f43486-02,
name=Build Dimension Dictionary, state=RUNNING}
 
java.io.IOException: java.lang.NullPointerException
 
at
org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71)
 
at
org.apache.kylin.dict.lookup.SnapshotTable.(SnapshotTable.java:64)
 
at
org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89)
 
at
org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208)
 
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59)
 
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
 
at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
 
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
 
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
 
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
 
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
 
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
 
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 
at java.lang.Thread.run(Thread.java:745)
 
Caused by: java.lang.NullPointerException
 
at
org.apache.kylin.engine.mr.HadoopUtil.fixWindowsPath(HadoopUtil.java:78)
 
at org.apache.kylin.engine.mr.HadoopUtil.makeURI(HadoopUtil.java:70)
 
at
org.apache.kylin.engine.mr.HadoopUtil.getFileSystem(HadoopUtil.java:65)
 
at
org.apache.kylin.engine.mr.DFSFileTable.getSizeAndLastModified(DFSFileTable.java:78)
 
at
org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:56)
 
... 16 more

[jira] [Created] (KYLIN-1584) Specify region cut size in cubedesc and leave the RealizationCapacity in model as a hint

Re: How to create a streaming model.

Re: No detail message when Error.

Re: No detail message when Error.

Re: ClassNotFoundException

Re: No detail message when Error.

No detail message when Error.

ClassNotFoundException

Re: build cube error

Re: build cube error

10 matches

Site Navigation

Mail list logo

Footer information