Hi,
When i build kylin cube , it failed at step 3 with error logs on webui
as follows:
Start to execute command:
com.kylinolap.job.hadoop.dict.CreateDictionaryJob -cubename
kylin_test11 -segmentname FULL_BUILD -input
/home/kylin/kylin-0175616e-50c9-44de-834d-404b1bca113a/kylin_test11/fact_distinct_columns
Command execute return code 2
And, logs from tomcat are as follows:
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:18,378][INFO][com.kylinolap.common.persistence.HBaseResourceStore.getTableName(HBaseResourceStore.java:108)]
- /job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name
project_metadata_job
java.io.IOException: Before,get putResource trace.
at
com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:171)
at com.kylinolap.job.JobDAO.writeJobResource(JobDAO.java:240)
at com.kylinolap.job.JobDAO.saveJob(JobDAO.java:161)
at com.kylinolap.job.JobDAO.updateJobInstance(JobDAO.java:199)
at
com.kylinolap.job.flow.JobFlowNode.updateJobStep(JobFlowNode.java:134)
at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:71)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:18,382][DEBUG][com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:176)]
- Saving resource /job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d (Store
project_metadata@hbase:182.118.45.55#182.118.45.56#182.118.45.57:2181:/hbase).oldTS:1435909715144,newTS:1435909758382
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:18,382][INFO][com.kylinolap.common.persistence.HBaseResourceStore.getTableName(HBaseResourceStore.java:108)]
- /job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name
project_metadata_job
java.io.IOException: After,get putResource trace.
at
com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:190)
at com.kylinolap.job.JobDAO.writeJobResource(JobDAO.java:240)
at com.kylinolap.job.JobDAO.saveJob(JobDAO.java:161)
at com.kylinolap.job.JobDAO.updateJobInstance(JobDAO.java:199)
at
com.kylinolap.job.flow.JobFlowNode.updateJobStep(JobFlowNode.java:134)
at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:71)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:18,390][DEBUG][com.kylinolap.job.cmd.JavaHadoopCmdOutput.appendOutput(JavaHadoopCmdOutput.java:96)]
- Start to execute command:
com.kylinolap.job.hadoop.dict.CreateDictionaryJob -cubename kylin_test9
-segmentname FULL_BUILD -input
/home/kylin/kylin-7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d/kylin_test9/fact_distinct_columns
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:18,398][INFO][com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:57)]
- Building snapshot of KYLIN_DIMENSION_RECHARGE_CHANNEL
usage: CreateDictionaryJob
-cubename <name> Cube name. For exmaple, flat_item_cube
-input <path> Input path
-segmentname <name> Cube segment name)
java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under
hdfs://namenode:9000/home/xitong/test-logsget-cube/KYLIN_DIMENSION_RECHARGE_CHANNEL,
but find 19
at com.kylinolap.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:119)
at
com.kylinolap.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:104)
at
com.kylinolap.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:79)
at com.kylinolap.dict.lookup.HiveTable.getFileTable(HiveTable.java:72)
at com.kylinolap.dict.lookup.HiveTable.getSignature(HiveTable.java:67)
at com.kylinolap.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:53)
at
com.kylinolap.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
at
com.kylinolap.cube.CubeManager.buildSnapshotTable(CubeManager.java:210)
at
com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:58)
at
com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:39)
at
com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:51)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:77)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:25,927][ERROR][com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:55)]
- Expect 1 and only 1 non-zero file under
hdfs://w-namenodefd2v.qss.zzbc2.qihoo.net:9000/home/xitong/dongtingting/test-logsget-cube/KYLIN_DIMENSION_RECHARGE_CHANNEL,
but find 19
java.lang.IllegalStateException: Expect 1 and only 1 non-zero file under
hdfs://namenode:9000/home/xitong/test-logsget-cube/KYLIN_DIMENSION_RECHARGE_CHANNEL,
but find 19
at com.kylinolap.dict.lookup.HiveTable.findOnlyFile(HiveTable.java:119)
at
com.kylinolap.dict.lookup.HiveTable.computeHDFSLocation(HiveTable.java:104)
at
com.kylinolap.dict.lookup.HiveTable.getHDFSLocation(HiveTable.java:79)
at com.kylinolap.dict.lookup.HiveTable.getFileTable(HiveTable.java:72)
at com.kylinolap.dict.lookup.HiveTable.getSignature(HiveTable.java:67)
at com.kylinolap.dict.lookup.SnapshotTable.<init>(SnapshotTable.java:53)
at
com.kylinolap.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:85)
at
com.kylinolap.cube.CubeManager.buildSnapshotTable(CubeManager.java:210)
at
com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:58)
at
com.kylinolap.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:39)
at
com.kylinolap.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:51)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.kylinolap.job.cmd.JavaHadoopCmd.execute(JavaHadoopCmd.java:53)
at com.kylinolap.job.flow.JobFlowNode.execute(JobFlowNode.java:77)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:25,928][DEBUG][com.kylinolap.job.cmd.JavaHadoopCmdOutput.appendOutput(JavaHadoopCmdOutput.java:96)]
- Command execute return code 2
[QuartzScheduler_Worker-9]:[2015-07-03
15:49:25,928][INFO][com.kylinolap.common.persistence.HBaseResourceStore.getTableName(HBaseResourceStore.java:108)]
- /job/7cd45ce1-b7cf-4c9d-a2fd-f8eecbbf900d getTableName Get Table name
project_metadata_job
It seems like i got too many file in that hive table, cause my
dimension tables were generated from another full_source_table by create
external table xxx location xxx and then insert overwrite table xxx select from
full_table. Thus it has generated many files out of my control.
Can you help me with this problem?
Thank you very much.
[email protected]
