[
https://issues.apache.org/jira/browse/KYLIN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433578#comment-17433578
]
mzz commented on KYLIN-5107:
----------------------------
Hi [~zhangyaqian]:
I try to use apache-kylin-3.1.0-bin-hadoop3.tar.gz,but if always report this
error
{code:java}
Error:
org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:193)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
... 4 more
Caused by: org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 41
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=3669205
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5177
HDFS: Number of bytes written=0
HDFS: Number of read operations=48
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
Job Counters
Failed reduce tasks=4
Killed map tasks=1
Launched map tasks=12
Launched reduce tasks=4
Data-local map tasks=12
Total time spent by all maps in occupied slots (ms)=2049760
Total time spent by all reduces in occupied slots (ms)=1404990
Total time spent by all map tasks (ms)=64055
Total time spent by all reduce tasks (ms)=10485
Total vcore-milliseconds taken by all map tasks=64055
Total vcore-milliseconds taken by all reduce tasks=10485
Total megabyte-milliseconds taken by all map tasks=65592320
Total megabyte-milliseconds taken by all reduce tasks=44959680
Map-Reduce Framework
Map input records=36
Map output records=72
Map output bytes=6179
Map output materialized bytes=1547
Input split bytes=2412
Combine input records=0
Spilled Records=72
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3593
CPU time spent (ms)=77860
Physical memory (bytes) snapshot=8037429248
Virtual memory (bytes) snapshot=36728225792
Total committed heap usage (bytes)=9890168832
Peak Map Physical memory (bytes)=679489536
Peak Map Virtual memory (bytes)=3069276160
File Input Format Counters
Bytes Read=2765
Job Diagnostics:Task failed task_1634097339477_0252_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:223)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
... 6 more
{code}
> when excuting the cube job at "Convert Cuboid Data to HFile" step：Error
> running child
> --------------------------------------------------------------------------------------
>
> Key: KYLIN-5107
> URL: https://issues.apache.org/jira/browse/KYLIN-5107
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v3.1.2
> Environment: hadoop3.2.1
> habse2.3.4
> Reporter: mzz
> Priority: Blocker
>
> As title, when excuting the cube job at "Convert Cuboid Data to HFile"
> step,throw an exception:
> {code:java}
> 2021-10-21 10:15:38,928 ERROR [main] org.apache.kylin.engine.mr.KylinReducer:
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureStoragePolicy(HFileOutputFormat3.java:468)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:287)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:243)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:559)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:141)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:135)
> at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> 2021-10-21 10:15:38,929 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 2773m
> 2021-10-21 10:15:38,929 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Total rows: 1
> 2021-10-21 10:15:38,929 ERROR [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.util.FSUtils.setStoragePolicy(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;Ljava/lang/String;)V
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureStoragePolicy(HFileOutputFormat3.java:468)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:287)
> at
> org.apache.kylin.storage.hbase.steps.HFileOutputFormat3$1.write(HFileOutputFormat3.java:243)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:559)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:141)
> at
> org.apache.kylin.storage.hbase.steps.CubeHFileJob$KeyValueReducer.doReduce(CubeHFileJob.java:135)
> at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1732)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
> I try move all of hbase libs to {KYLIN_HOME}/lib ,but this problem not
> solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
