Hi friend, I saw log contains "hadoop.fs.FileStatus.isErasureCoded", which indicated that you are using a Hadoop3 env, is it CDH6? I guess the error maybe caused by some mistake in HDFS configuration. I also saw other information which seems be helpful, did you set the correct value for dfs.encryption.key.provider.uri?
I think you should copy the SQL generated by Kylin and pasted them into Hive CLI, and change your configuration with Hive and HDFS to ensure sql can be executed successfully. ---------------- Best wishes, Xiaoxiang Yu 发件人: 欧阳苗 <[email protected]> 答复: "[email protected]" <[email protected]> 日期: 2019年12月9日 星期一 18:15 收件人: user <[email protected]> 主题: 构建cube的第一步到了如下错误, 版本 apache-kylin-2.6.4-bin-cdh60.tar.gz java.io.IOException: OS command error exit with return code: 155, error message: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/modules/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false OK Time taken: 2.513 seconds OK Time taken: 0.385 seconds OK Time taken: 0.529 seconds OK Time taken: 0.521 seconds Query ID = root_20191208010521_cb1bca53-9910-4ab1-ac6e-a4de6dcdff39 Total jobs = 1 Execution completed successfully MapredLocal task succeeded Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator 19/12/08 01:05:42 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication 19/12/08 01:05:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm164 19/12/08 01:05:44 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/12/08 01:05:49 INFO input.FileInputFormat: Total input paths to process : 1 19/12/08 01:05:49 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0 19/12/08 01:05:49 INFO mapreduce.JobSubmitter: number of splits:1 19/12/08 01:05:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575269010295_0006 19/12/08 01:05:50 INFO impl.YarnClientImpl: Submitted application application_1575269010295_0006 19/12/08 01:05:50 INFO mapreduce.Job: The url to track the job: http://miao2.com:8088/proxy/application_1575269010295_0006/ Starting Job = job_1575269010295_0006, Tracking URL = http://miao2.com:8088/proxy/application_1575269010295_0006/ Kill Command = /opt/modules/hadoop-2.7.7/bin/hadoop job -kill job_1575269010295_0006 Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0 19/12/08 01:06:05 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2019-12-08 01:06:05,920 Stage-4 map = 0%, reduce = 0% 19/12/08 01:06:19 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:19 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. 2019-12-08 01:06:19,383 Stage-4 map = 100%, reduce = 0%, Cumulative CPU 2.52 sec 19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FileSystemCounterGroup: HDFS_BYTES_READ_EC is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_PHYSICAL_MEMORY_BYTES_MAX is not a recognized counter. 19/12/08 01:06:20 WARN counters.FrameworkCounterGroup: MAP_VIRTUAL_MEMORY_BYTES_MAX is not a recognized counter. MapReduce Total cumulative CPU time: 2 seconds 520 msec Ended Job = job_1575269010295_0006 Loading data to table default.kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 19/12/08 01:06:20 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !! FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. org.apache.hadoop.fs.FileStatus.isErasureCoded()Z MapReduce Jobs Launched: Stage-Stage-4: Map: 1 Cumulative CPU: 2.52 sec HDFS Read: 8969 HDFS Write: 551 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 520 msec The command is: hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 ( \`EMP_JOB\` string ,\`EMP_HIREDATE\` string ,\`EMP_DEPTNO\` int ,\`DEPT_DNAME\` string ,\`DEPT_LOC\` int ,\`EMP_SAL\` double ) STORED AS SEQUENCEFILE LOCATION 'hdfs://nameservice1/kylin/kylin_metadata/kylin-99121fe3-9dce-cdf0-f684-d97cb998c629/kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135'; ALTER TABLE kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135 SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE \`kylin_intermediate_emp_cube_da04d657_54f0_d436_1e6f_9b5c1d48d135\` SELECT \`EMP\`.\`JOB\` as \`EMP_JOB\` ,\`EMP\`.\`HIREDATE\` as \`EMP_HIREDATE\` ,\`EMP\`.\`DEPTNO\` as \`EMP_DEPTNO\` ,\`DEPT\`.\`DNAME\` as \`DEPT_DNAME\` ,\`DEPT\`.\`LOC\` as \`DEPT_LOC\` ,\`EMP\`.\`SAL\` as \`EMP_SAL\` FROM \`DEFAULT\`.\`EMP\` as \`EMP\` INNER JOIN \`DEFAULT\`.\`DEPT\` as \`DEPT\` ON \`EMP\`.\`DEPTNO\` = \`DEPT\`.\`DEPTNO\` WHERE 1=1; " --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96) at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:62) at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:99) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
