we want to run apache kylin on spark, we got this at #7 Step Name: Build Cube with Spark
OS command error exit with return code: 1, error message: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227) at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161) at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:153) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1226) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219) ... 18 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) when we try to add hadoop-yarn-common-2.7.3.2.6.1.0-129.jar to ${KYLIN_HOME}/spark/jars to solve problem above , then we got another problem at #7 Step Name: Build Cube with Spark: OS command error exit with return code: 1, error message: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider at org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75) at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163) at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:153) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1226) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) The command is: export HADOOP_CONF_DIR=/mnt/kylin/hadoop-conf && /mnt/kylin/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=10 --conf spark.yarn.queue=kylin --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.driver.extraJavaOptions=-Dhdp.version=current --conf spark.master=yarn --conf spark.executor.extraJavaOptions=-Dhdp.version=current --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spark.executor.memory=13G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.executor.cores=6 --conf spark.submit.deployMode=cluster --jars /usr/hdp/2.6.1.0-129/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.6.1.0-129/hbase/lib/metrics-core-2.2.0.jar,/usr/hdp/2.6.1.0-129/hbase/lib/guava-12.0.1.jar, /mnt/kylin/lib/kylin-job-2.3.1.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_spark_3f06753c_7cd9_4898_9e8c_32d987004b64 -output hdfs://hdfs-ha/kylin/kylin_metadata/kylin-d67bba33-2e73-4a70-9e6f-7a907bb86715/kylin_spark/cuboid/ -segmentId 3f06753c-7cd9-4898-9e8c-32d987004b64 -metaUrl kylin_metadata@hdfs,path=hdfs://hdfs-ha/kylin/kylin_metadata/metadata/3f06753c-7cd9-4898-9e8c-32d987004b64 -cubename kylin_spark envirment: apache-kylin-2.3.1-bin-hbase1x.tar.gz HDP-2.6.1.0 Current HDFS 2.7.3 YARN 2.7.3 MapReduce2 2.7.3 Tez 0.7.0 Hive 1.2.1000 HBase 1.1.2 ZooKeeper 3.4.6 Storm 1.1.0 Ambari Metrics 0.1.0 Kafka 0.10.1 Spark2 2.x Elasticsearch 5.6.3 here is kylin.propertys : kylin.env.hdfs-working-dir=/kylin kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx18g kylin.engine.mr.config-override.mapreduce.map.memory.mb=9216 #kylin.engine.mr.config-override.mapreduce.map.memory.mb=9216 kylin.storage.hbase.compression-codec=snappy kylin.job.sampling-percentage=30 kylin.engine.mr.reduce-input-mb=800 kylin.engine.mr.max-reducer-number=800 kylin.engine.mr.uhc-reducer-count=5 kylin.engine.spark-conf.spark.master=yarn kylin.engine.spark-conf.spark.submit.deployMode=cluster kylin.engine.spark-conf.spark.executor.memory=3G kylin.engine.spark-conf.spark.executor.cores=6 kylin.engine.spark-conf.spark.executor.instances=10 kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp.version=current kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dhdp.version=current Please advise what we can do to solve above problems thanks [email protected] phone 18614221005
