Kylin does not support reading dimension table from multiple files(it assumes that dimension table is small). So you may have to merge them manually.
BTW, it seems that the you are using old version of Kylin, we have make our first apache release lately. We strongly recommend you using the latest release(via www.kylin.io) Best Regard Zhou QianHao On 7/3/15, 5:13 PM, "[email protected]" <[email protected]> wrote: >Hi, > When i build kylin cube , it failed at step 3 with error logs on >webui as follows: > Start to execute command: > com.kylinolap.job.hadoop.dict.CreateDictionaryJob -cubename >kylin_test11 -segmentname FULL_BUILD -input >/home/kylin/kylin-0175616e-50c9-44de-834d-404b1bca113a/kylin_test11/fact_d >istinct_columns > Command execute return code 2 > > And, logs from tomcat are as follows: > [QuartzScheduler_Worker-9]:[2015-07-03 >15:49:18,378][INFO][com.kylinolap.common.persistence.HBaseResourceStore.ge >tTableName(HBaseResourceStore.java:108)] - >/job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name >project_metadata_job >java.io.IOException: Before,get putResource trace. > at >com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.j >ava:171) > at com.kylinolap.job.JobDAO.writeJobResource(JobDAO.java:240) > at com.kylinolap.job.JobDAO.saveJob(JobDAO.java:161) > at com.kylinolap.job.JobDAO.updateJobInstance(JobDAO.java:199) > at >com.kylinolap.job.flow.JobFlowNode.updateJobStep(JobFlowNode.java:134) > at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:71) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at >org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:5 >73) >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:18,382][DEBUG][com.kylinolap.common.persistence.ResourceStore.putRes >ource(ResourceStore.java:176)] - Saving resource >/job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d (Store >project_metadata@hbase:182.118.45.55#182.118.45.56#182.118.45.57:2181:/hba >se).oldTS:1435909715144,newTS:1435909758382 >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:18,382][INFO][com.kylinolap.common.persistence.HBaseResourceStore.ge >tTableName(HBaseResourceStore.java:108)] - >/job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name >project_metadata_job >java.io.IOException: After,get putResource trace. > at >com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.j >ava:190) > at com.kylinolap.job.JobDAO.writeJobResource(JobDAO.java:240) > at com.kylinolap.job.JobDAO.saveJob(JobDAO.java:161) > at com.kylinolap.job.JobDAO.updateJobInstance(JobDAO.java:199) > at >com.kylinolap.job.flow.JobFlowNode.updateJobStep(JobFlowNode.java:134) > at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:71) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at >org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:5 >73) >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:18,390][DEBUG][com.kylinolap.job.cmd.JavaHadoopCmdOutput.appendOutpu >t(JavaHadoopCmdOutput.java:96)] - Start to execute command: >com.kylinolap.job.hadoop.dict.CreateDictionaryJob -cubename kylin_test9 >-segmentname FULL_BUILD -input >/home/kylin/kylin-7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d/kylin_test9/fact_di >stinct_columns >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:18,398][INFO][com.kylinolap.cube.cli.DictionaryGeneratorCLI.processS >egment(DictionaryGeneratorCLI.java:57)] - Building snapshot of >KYLIN_DIMENSION_RECHARGE_CHANNEL >usage: CreateDictionaryJob > -cubename <name> Cube name. For exmaple, flat_item_cube > -input <path> Input path > -segmentname <name> Cube segment name) >java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under >hdfs://namenode:9000/home/xitong/test-logsget-cube/KYLIN_DIMENSION_RECHARG >E_CHANNEL, but find 19 > at >com.kylinolap.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:119) > at >com.kylinolap.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:104 >) > at >com.kylinolap.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:79) > at >com.kylinolap.dict.lookup.HiveTable.getFileTable(HiveTable.java:72) > at >com.kylinolap.dict.lookup.HiveTable.getSignature(HiveTable.java:67) > at >com.kylinolap.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:53) > at >com.kylinolap.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.ja >va:85) > at >com.kylinolap.cube.CubeManager.buildSnapshotTable(CubeManager.java:210) > at >com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGen >eratorCLI.java:58) > at >com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGen >eratorCLI.java:39) > at >com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob. >java:51) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at >com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53) > at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:77) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at >org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:5 >73) >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:25,927][ERROR][com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run >(CreateDictionaryJob.java:55)] - Expect 1 and only 1 non-zero file under >hdfs://w-namenodefd2v.qss.zzbc2.qihoo.net:9000/home/xitong/dongtingting/te >st-logsget-cube/KYLIN_DIMENSION_RECHARGE_CHANNEL, but find 19 >java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under >hdfs://namenode:9000/home/xitong/test-logsget-cube/KYLIN_DIMENSION_RECHARG >E_CHANNEL, but find 19 > at >com.kylinolap.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:119) > at >com.kylinolap.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:104 >) > at >com.kylinolap.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:79) > at >com.kylinolap.dict.lookup.HiveTable.getFileTable(HiveTable.java:72) > at >com.kylinolap.dict.lookup.HiveTable.getSignature(HiveTable.java:67) > at >com.kylinolap.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:53) > at >com.kylinolap.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.ja >va:85) > at >com.kylinolap.cube.CubeManager.buildSnapshotTable(CubeManager.java:210) > at >com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGen >eratorCLI.java:58) > at >com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGen >eratorCLI.java:39) > at >com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob. >java:51) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at >com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53) > at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:77) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at >org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:5 >73) >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:25,928][DEBUG][com.kylinolap.job.cmd.JavaHadoopCmdOutput.appendOutpu >t(JavaHadoopCmdOutput.java:96)] - Command execute return code 2 >[QuartzScheduler_Worker-9]:[2015-07-03 >15:49:25,928][INFO][com.kylinolap.common.persistence.HBaseResourceStore.ge >tTableName(HBaseResourceStore.java:108)] - >/job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name >project_metadata_job > > It seems like i got too many file in that hive table, cause my >dimension tables were generated from another full_source_table by create >external table xxx location xxx and then insert overwrite table xxx >select from full_table. Thus it has generated many files out of my >control. > Can you help me with this problem? > Thank you very much. > > > >[email protected]
