Hi Rupeng, Here's the hadoop version: ----------- hadoop version Hadoop 2.7.2 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41 Compiled by jenkins on 2016-01-26T00:08Z Compiled with protoc 2.5.0 >From source with checksum d0fda26633fa762bff87ec759ebe689c This command was run using /usr/local/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar ------------
Thanks, Liang On Thu, Dec 12, 2019 at 7:15 PM Wang rupeng <[email protected]> wrote: > Hi, > > It seems like environment problem. What’s your cdh or hdp version? > > > > > > ------------------- > > Best wishes, > > Rupeng Wang > > > > > > *发件人**: *liang <[email protected]> > *答复**: *<[email protected]> > *日期**: *2019年12月12日 星期四 15:46 > *收件人**: *<[email protected]> > *主题**: *Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile > > > > Hi there, > > > > I'm trying to build cube with Spark. I've followed the instructions in > this tutorial: Build Cube with Spark > <http://kylin.apache.org/docs/tutorial/cube_spark.html>. But the job > failed at step: "Convert Cuboid Data to HFile". The logs shows > "ClassNotFoundException: > org.apache.hadoop.hbase.io.MetricsIOWrapper" > > > > org.apache.kylin.engine.spark.exception.SparkException: OS command error > exit with return code: 1, error message: 2019-12-12 15:10:46 WARN > SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead' > has been deprecated as of Spark 2.3 and may be removed in the future. > Please use the new key 'spark.executor.memoryOverhead' instead. > 2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at > host/10.16.15.15:9032 > 2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from > cluster with 150 NodeManagers > 2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not > requested more than the maximum memory capability of the cluster (49150 MB > per container) > 2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with > 2432 MB memory including 384 MB overhead > 2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context > for our AM > 2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment > for our AM container > 2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM > container > 2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems > are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar > 2019-12-12 15:10:50 INFO Client:54 - Uploading resource > file:/usr/local/kylin/lib/kylin-job-2.6.4.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar > 2019-12-12 15:10:51 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar > 2019-12-12 15:10:51 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar > 2019-12-12 15:10:51 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar > 2019-12-12 15:10:51 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar > 2019-12-12 15:10:51 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar > 2019-12-12 15:10:52 INFO Client:54 - Uploading resource > file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar > 2019-12-12 15:10:52 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar > 2019-12-12 15:10:52 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar > 2019-12-12 15:10:52 WARN Client:66 - Same path resource > file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added > multiple times to distributed cache. > 2019-12-12 15:10:52 INFO Client:54 - Uploading resource > file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar > 2019-12-12 15:10:53 INFO Client:54 - Uploading resource > file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip > -> > hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip > 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to: > hadoop > 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to: > hadoop > 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups > to: > 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls groups > to: > 2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hadoop); groups with view permissions: Set(); users with modify > permissions: Set(hadoop); groups with modify permissions: Set() > 2019-12-12 15:10:54 INFO Client:54 - Submitting application > application_1549925837808_10474661 to ResourceManager > 2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application > application_1549925837808_10474661 > 2019-12-12 15:10:55 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:10:55 INFO Client:54 - > client token: N/A > diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the > scheduler and is not yet activated. (Resource request: <memory:3072, > vCores:1> exceeds current queue or its parents maximum resource allowed). > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.hadoop > start time: 1576134654497 > final status: UNDEFINED > tracking URL: > http://host.example.com:8981/proxy/application_1549925837808_10474661/ > user: hadoop > 2019-12-12 15:10:56 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:10:57 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:10:58 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:10:59 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:00 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:01 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:02 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:03 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:04 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:05 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:06 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:07 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:08 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:09 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:10 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:11 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:11 INFO Client:54 - > client token: N/A > diagnostics: N/A > ApplicationMaster host: 10.16.15.244 > ApplicationMaster RPC port: 0 > queue: root.hadoop > start time: 1576134654497 > final status: UNDEFINED > tracking URL: > http://host.example.com:8981/proxy/application_1549925837808_10474661/ > user: hadoop > 2019-12-12 15:11:12 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:13 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:14 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:15 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:16 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:17 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:18 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:19 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:20 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:21 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:22 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:23 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:24 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:25 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:26 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:27 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:28 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:29 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:30 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:31 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:32 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:32 INFO Client:54 - > client token: N/A > diagnostics: AM container is launched, waiting for AM container to > Register with RM > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.hadoop > start time: 1576134654497 > final status: UNDEFINED > tracking URL: > http://host.example.com:8981/proxy/application_1549925837808_10474661/ > user: hadoop > 2019-12-12 15:11:33 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:34 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:35 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:36 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:37 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:38 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:39 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:40 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:41 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: ACCEPTED) > 2019-12-12 15:11:42 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:42 INFO Client:54 - > client token: N/A > diagnostics: N/A > ApplicationMaster host: 10.16.15.59 > ApplicationMaster RPC port: 0 > queue: root.hadoop > start time: 1576134654497 > final status: UNDEFINED > tracking URL: > http://host.example.com:8981/proxy/application_1549925837808_10474661/ > user: hadoop > 2019-12-12 15:11:43 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:44 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:45 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:46 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:47 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:48 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:49 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:50 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:51 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:52 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:53 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:54 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:55 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:56 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:57 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:58 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:11:59 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:00 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:01 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:02 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:03 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:04 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:05 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:06 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:07 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:08 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: RUNNING) > 2019-12-12 15:12:09 INFO Client:54 - Application report for > application_1549925837808_10474661 (state: FINISHED) > 2019-12-12 15:12:09 INFO Client:54 - > client token: N/A > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: > Job aborted. > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > Caused by: org.apache.spark.SparkException: Job aborted. > at > org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) > at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) > at > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081) > at > org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831) > at > org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 6 more > Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task > 0.3 in stage 1.0 (TID 5, host.dns.example.com, executor 15): > org.apache.spark.SparkException: Task failed while writing rows > at > org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/io/MetricsIOWrapper > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) > at > org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415) > at > org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139) > ... 8 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.io.MetricsIOWrapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 16 more > > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831) > at scala.Option.foreach(Option.scala:257) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087) > at > org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78) > ... 16 more > Caused by: org.apache.spark.SparkException: Task failed while writing rows > at > org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/io/MetricsIOWrapper > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) > at > org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130) > at > org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415) > at > org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139) > ... 8 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.io.MetricsIOWrapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 16 more > > ApplicationMaster host: 10.16.15.59 > ApplicationMaster RPC port: 0 > queue: root.hadoop > start time: 1576134654497 > final status: FAILED > tracking URL: > http://host.example.com:8981/proxy/application_1549925837808_10474661/ > user: hadoop > Exception in thread "main" org.apache.spark.SparkException: Application > application_1549925837808_10474661 finished with failed status > at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165) > at > org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called > 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory > /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409 > 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory > /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1 > The command is: > export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop && > /usr/local/spark/bin/spark-submit --class > org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40 > --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf > spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf > spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec > --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn > --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf > spark.hadoop.yarn.timeline-service.enabled=false --conf > spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf > spark.eventLog.dir=hdfs:///kylin/spark-history --conf > spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1 > --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false > --conf spark.yarn.executor.memoryOverhead=1024 --conf > spark.hadoop.dfs.replication=2 --conf > spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G > --conf spark.submit.deployMode=cluster --conf > spark.shuffle.service.enabled=true --jars > /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar, > /usr/local/kylin/lib/kylin-job-2.6.4.jar -className > org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions > hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile > -counterOutput > hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter > -cubename test1 -output > hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile > -input > hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/ > -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl > kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata > -hbaseConfPath > hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > This error message is very like the tips on the tutorial which I mentioned > above. I've tried to put hbase-hadoop-compat-1.1.5.jar and > hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I > found there's no spark folder under the $KYLIN_HOME. This is strange > because the tutorial said Kylin has a built-in Spark. Maybe the docs are > out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put > these two jars into the "jars" folder. But when after I resume the job, > that error happened again. Nothing helps. > I've also tried to put these two jars under the $SPARK_HOME/jars. But it > still doesn't work. > > > > These versions are my current setup: > > Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz > > Spark: 2.3.2 > > HBase: 1.1.5 > > Hadoop: 2.7.2 > > > > Thanks, > > Liang > > > > >
