[jira] [Created] (KYLIN-1584) Specify region cut size in cubedesc and leave the RealizationCapacity in model as a hint
hongbin ma created KYLIN-1584: - Summary: Specify region cut size in cubedesc and leave the RealizationCapacity in model as a hint Key: KYLIN-1584 URL: https://issues.apache.org/jira/browse/KYLIN-1584 Project: Kylin Issue Type: Improvement Reporter: hongbin ma Assignee: hongbin ma currently the cube region's size is determined in model's capacity(RealizationCapacity), it will enforce different realizations under the same model having uniformed region size. This is not wanted feature because cube builder might need to try out different region size to tune performance. I'm suggesting to add a field called "region_size" (float type, 1.0 means 1G per region) in cube desc to explicitly indicate the cube's region size. The field "region_size" will not be included in cubedesc's signature calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: How to create a streaming model.
It's hard to describe all steps in few words.. we owe a tutorial for this, let me open a task for this, please watch it for any update: https://issues.apache.org/jira/browse/KYLIN-1582 2016-04-13 19:39 GMT+08:00 陈佛林: > There is no example or guide to tell me how to create a streaming model. > > I don't know how to configure kafka topics,is it just a simple string(such > as "topic1")? > and where to configure kafka address > -- Best regards, Shaofeng Shi
Re: No detail message when Error.
1.5.0 2016-04-13 19:41 GMT+08:00 hongbin ma: > what is the kylin version? > > On Wed, Apr 13, 2016 at 7:02 PM, 陈佛林 wrote: > > > when i delete a model, it got "Failed to take action", got a "500" > code.but > > the model has been deleted at backend. > > > > > > all logs when i delete a model: > > > > 2016-04-13 19:00:00,449 INFO [http-bio-7070-exec-7] > > invertedindex.IIDescManager: Initializing IIDescManager with config > > kylin_metadata@hbase > > 2016-04-13 19:00:00,449 INFO [http-bio-7070-exec-7] > > invertedindex.IIDescManager: Reloading all II desc from folder > > kylin_metadata(key='/invertedindex_desc')@kylin_metadata@hbase > > 2016-04-13 19:00:00,456 INFO [http-bio-7070-exec-7] > > metadata.MetadataManager: Dropping model 'dd' > > 2016-04-13 19:00:00,463 INFO [pool-8-thread-1] restclient.Broadcaster: > new > > broadcast event:BroadcastEvent{type=project, name=myproject, > action=update} > > 2016-04-13 19:00:00,483 INFO [pool-8-thread-1] restclient.Broadcaster: > new > > broadcast event:BroadcastEvent{type=data_model, name=dd, action=drop} > > 2016-04-13 19:00:00,487 INFO [http-bio-7070-exec-8] > > controller.CacheController: wipe cache type: PROJECT event:UPDATE > > name:myproject > > 2016-04-13 19:00:00,487 INFO [http-bio-7070-exec-8] > service.CacheService: > > rebuild cache type: PROJECT name:myproject > > 2016-04-13 19:00:00,490 INFO [http-bio-7070-exec-8] > service.CacheService: > > removeOLAPDataSource is called for project myproject > > 2016-04-13 19:00:00,495 INFO [http-bio-7070-exec-8] > > controller.CacheController: wipe cache type: DATA_MODEL event:DROP > name:dd > > > > 2016-04-13 18:26 GMT+08:00 hongbin ma : > > > > > have you checked $KYLIN_HOME/logs/kylin.log? > > > > > > On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林 wrote: > > > > > > > hi, > > > > > > > > almost all error just show a little message. > > > > > > > > such as: > > > > > > > > Error Message > > > > > > > > Failed to take action. > > > > > > > > > > > > I have no idea what happened. > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > *Bin Mahone | 马洪宾* > > > Apache Kylin: http://kylin.io > > > Github: https://github.com/binmahone > > > > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone >
Re: No detail message when Error.
when i delete a model, it got "Failed to take action", got a "500" code.but the model has been deleted at backend. all logs when i delete a model: 2016-04-13 19:00:00,449 INFO [http-bio-7070-exec-7] invertedindex.IIDescManager: Initializing IIDescManager with config kylin_metadata@hbase 2016-04-13 19:00:00,449 INFO [http-bio-7070-exec-7] invertedindex.IIDescManager: Reloading all II desc from folder kylin_metadata(key='/invertedindex_desc')@kylin_metadata@hbase 2016-04-13 19:00:00,456 INFO [http-bio-7070-exec-7] metadata.MetadataManager: Dropping model 'dd' 2016-04-13 19:00:00,463 INFO [pool-8-thread-1] restclient.Broadcaster: new broadcast event:BroadcastEvent{type=project, name=myproject, action=update} 2016-04-13 19:00:00,483 INFO [pool-8-thread-1] restclient.Broadcaster: new broadcast event:BroadcastEvent{type=data_model, name=dd, action=drop} 2016-04-13 19:00:00,487 INFO [http-bio-7070-exec-8] controller.CacheController: wipe cache type: PROJECT event:UPDATE name:myproject 2016-04-13 19:00:00,487 INFO [http-bio-7070-exec-8] service.CacheService: rebuild cache type: PROJECT name:myproject 2016-04-13 19:00:00,490 INFO [http-bio-7070-exec-8] service.CacheService: removeOLAPDataSource is called for project myproject 2016-04-13 19:00:00,495 INFO [http-bio-7070-exec-8] controller.CacheController: wipe cache type: DATA_MODEL event:DROP name:dd 2016-04-13 18:26 GMT+08:00 hongbin ma: > have you checked $KYLIN_HOME/logs/kylin.log? > > On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林 wrote: > > > hi, > > > > almost all error just show a little message. > > > > such as: > > > > Error Message > > > > Failed to take action. > > > > > > I have no idea what happened. > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone >
Re: ClassNotFoundException
are you using CHD, kylin 1.5.0? take a look at http://apache-kylin.74782.x6.nabble.com/Stuck-with-ClassNotFoundException-HCatInputFormat-td4032.html it can be found if you search google On Wed, Apr 13, 2016 at 6:01 PM, 耳东 <775620...@qq.com> wrote: > Hi all: when I built the cube, this exception occured. Is there any > solution to this excepiton. I am using Kylin on a cluster. > > > > > Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) at > org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1776) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.lang.ClassNotFoundException: Class > org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) > ... 8 more -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
Re: No detail message when Error.
have you checked $KYLIN_HOME/logs/kylin.log? On Wed, Apr 13, 2016 at 6:18 PM, 陈佛林wrote: > hi, > > almost all error just show a little message. > > such as: > > Error Message > > Failed to take action. > > > I have no idea what happened. > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
No detail message when Error.
hi, almost all error just show a little message. such as: Error Message Failed to take action. I have no idea what happened.
ClassNotFoundException
Hi all?? when I built the cube, this exception occured. Is there any solution to this excepiton. I am using Kylin on a cluster. Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1776) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) ... 8 more
Re: build cube error
Thanks. When i replace the hive view with other tables, every thing is ok. 2016-04-13 17:02 GMT+08:00 wangxianbin1...@gmail.com < wangxianbin1...@gmail.com>: > hi! > > what do you mean? are you using hive view as lookup table? > > if it is so, it is a known issue, check this out, > https://issues.apache.org/jira/browse/KYLIN-1077 > > best regards! > > > > wangxianbin1...@gmail.com > > From: 陈佛林 > Date: 2016-04-13 16:51 > To: dev > Subject: build cube error > all job at yarn are SUCCESS. > > > 2016-04-13 15:54:58,765 WARN [http-bio-7070-exec-1] > service.CacheService:108 : skip cleaning cache for > 668468ba-f470-4c37-9b12-d66aa44cef30 > > 2016-04-13 15:54:58,903 ERROR [pool-5-thread-6] > execution.AbstractExecutable:62 : error execute > HadoopShellExecutable{id=4e9d2b2a-4f20-4b35-9dc6-bf2185f43486-02, > name=Build Dimension Dictionary, state=RUNNING} > > java.io.IOException: java.lang.NullPointerException > > at > org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71) > > at > org.apache.kylin.dict.lookup.SnapshotTable.(SnapshotTable.java:64) > > at > > org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89) > > at > org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208) > > at > > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59) > > at > > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) > > at > > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > at > > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60) > > at > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > > at > > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > > at > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > > at > > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.NullPointerException > > at > org.apache.kylin.engine.mr.HadoopUtil.fixWindowsPath(HadoopUtil.java:78) > > at > org.apache.kylin.engine.mr.HadoopUtil.makeURI(HadoopUtil.java:70) > > at > org.apache.kylin.engine.mr.HadoopUtil.getFileSystem(HadoopUtil.java:65) > > at > > org.apache.kylin.engine.mr.DFSFileTable.getSizeAndLastModified(DFSFileTable.java:78) > > at > org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:56) > > ... 16 more >
Re: build cube error
hi! what do you mean? are you using hive view as lookup table? if it is so, it is a known issue, check this out, https://issues.apache.org/jira/browse/KYLIN-1077 best regards! wangxianbin1...@gmail.com From: 陈佛林 Date: 2016-04-13 16:51 To: dev Subject: build cube error all job at yarn are SUCCESS. 2016-04-13 15:54:58,765 WARN [http-bio-7070-exec-1] service.CacheService:108 : skip cleaning cache for 668468ba-f470-4c37-9b12-d66aa44cef30 2016-04-13 15:54:58,903 ERROR [pool-5-thread-6] execution.AbstractExecutable:62 : error execute HadoopShellExecutable{id=4e9d2b2a-4f20-4b35-9dc6-bf2185f43486-02, name=Build Dimension Dictionary, state=RUNNING} java.io.IOException: java.lang.NullPointerException at org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:71) at org.apache.kylin.dict.lookup.SnapshotTable.(SnapshotTable.java:64) at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:89) at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:208) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.kylin.engine.mr.HadoopUtil.fixWindowsPath(HadoopUtil.java:78) at org.apache.kylin.engine.mr.HadoopUtil.makeURI(HadoopUtil.java:70) at org.apache.kylin.engine.mr.HadoopUtil.getFileSystem(HadoopUtil.java:65) at org.apache.kylin.engine.mr.DFSFileTable.getSizeAndLastModified(DFSFileTable.java:78) at org.apache.kylin.source.hive.HiveTable.getSignature(HiveTable.java:56) ... 16 more