[ https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698748#comment-16698748 ]
Yiping Wei commented on KYLIN-3607: ----------------------------------- I am facing the same problem on Kylin-2.5.1 with HDP-2.6.5.0-292. I am sure the hadoop compatibility jar is on the classpath.This is the command in my build #7 step log: {code:bash} export HADOOP_CONF_DIR=/usr/hdp/2.6.5.0-292/hadoop/conf && /usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=3 --conf spark.yarn.archive=hdfs://slave1:8020/kylin/spark/spark-libs.jar --conf spark.yarn.queue=default --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.driver.extraJavaOptions=-Dhdp.version=current --conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec --conf spark.master=yarn --conf spark.executor.extraJavaOptions=-Dhdp.version=current --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spark.executor.memory=2G --conf spark.eventLog.enabled=false --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.driver.memory=1G --conf spark.submit.deployMode=client --conf spark.shuffle.service.enabled=true --jars /usr/hdp/2.6.5.0-292/hbase/lib/hbase-common-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-server-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-client-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-protocol-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-hadoop-compat-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.6.5.0-292/hbase/lib/metrics-core-2.2.0.jar, /home/kylin/apache-kylin-2.5.1-bin-hbase1x/lib/kylin-job-2.5.1.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/rowkey_stats/part-r-00000_hfile -counterOutput hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/counter -cubename INJECTION_MOLDING_COUNT -output hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/hfile -input hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/cuboid/ -segmentId ab60256a-430d-d3c7-f2e3-98daed21717b -metaUrl kylin_metadata@hdfs,path=hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/metadata -hbaseConfPath hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/hbase-conf.xml {code} > can't build cube with spark in v2.5.0 > ------------------------------------- > > Key: KYLIN-3607 > URL: https://issues.apache.org/jira/browse/KYLIN-3607 > Project: Kylin > Issue Type: Bug > Reporter: ANIL KUMAR > Priority: Major > > in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, > the following is the related exception: > > ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted > due to stage failure: Task 0 in stage 1.0 failed 4 times, > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192) > ... 15 more > Caused by: java.util.NoSuchElementException > at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)