[ https://issues.apache.org/jira/browse/KYLIN-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174300#comment-17174300 ]
wangrupeng commented on KYLIN-4690: ----------------------------------- Could you provide more information about your environment? I test it in my CDH5.7 with cluster mode and it works fine. > BUILD CUBE - job fail on spark clusters mode - #7 Step Name: Build Cube with > Spark > ----------------------------------------------------------------------------------- > > Key: KYLIN-4690 > URL: https://issues.apache.org/jira/browse/KYLIN-4690 > Project: Kylin > Issue Type: Bug > Components: Spark Engine > Affects Versions: v3.1.0 > Reporter: James > Priority: Critical > > BUILD CUBE - job fail on spark clusters mode - #7 Step Name: Build Cube with > Spark > Executor: > export > HADOOP_CONF_DIR=/app/kylin/apache-kylin-3.1.0-bin-hbase1x/kylin_hadoop_conf_dir > && /usr/hdp/current/spark2-client/bin/spark-submit --class > org.apache.kylin.common.util.SparkEntry --name "Build Cube with > Spark:CBE_DEV[20200102000000_20200103000000]" --conf spark.executor.cores=5 > --conf spark.hadoop.yarn.timeline-service.enabled=false --conf > spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec > --conf spark.executor.memoryOverhead=1024 --conf > spark.executor.extraJavaOptions=-Dhdp.version=2.6.4.149-3 --conf > spark.master=yarn --conf > spark.hadoop.mapreduce.output.fileoutputformat.compress=true --conf > spark.executor.instances=5 --conf > spark.kryo.register=org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage > --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=2.6.4.149-3 --conf > spark.executor.memory=4G --conf spark.yarn.queue=sgz1-criskapp-haas_dev > --conf spark.submit.deployMode=cluster --conf > spark.dynamicAllocation.minExecutors=0 --conf spark.network.timeout=600 > --conf spark.hadoop.dfs.replication=2 --conf > spark.yarn.executor.memoryOverhead=1024 --conf > spark.dynamicAllocation.executorIdleTimeout=300 --conf > spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf > spark.driver.memory=5G --conf > spark.driver.extraJavaOptions=-Dhdp.version=2.6.4.149-3 --conf > spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec --conf > spark.eventLog.enabled=true --conf spark.shuffle.service.enabled=true > --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf > spark.dynamicAllocation.maxExecutors=15 --conf > spark.dynamicAllocation.enabled=true --jars > /app/kylin/apache-kylin-3.1.0-bin-hbase1x/lib/kylin-job-3.1.0.jar > /app/kylin/apache-kylin-3.1.0-bin-hbase1x/lib/kylin-job-3.1.0.jar -className > org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable > kylin310.kylin_intermediate_cbe_dev_02f32a29_1d51_0cb0_37ba_825333d38c8d > -output > hdfs://<IP:PORT>/dev/kylin310/kylin-0f5b105d-4794-e7ce-b329-fd7a83cb1aa2/CBE_DEV/cuboid/ > -input > hdfs://<IP:PORT>/dev/kylin310/kylin-0f5b105d-4794-e7ce-b329-fd7a83cb1aa2/kylin_intermediate_cbe_dev_02f32a29_1d51_0cb0_37ba_825333d38c8d > -segmentId 02f32a29-1d51-0cb0-37ba-825333d38c8d -metaUrl > crr_kylin_dev240@hdfs,path=hdfs://<IP:PORT>/dev/kylin310/kylin-0f5b105d-4794-e7ce-b329-fd7a83cb1aa2/CBE_DEV/metadata > -cubename CBE_DEV > > Step Name: > #7 Step Name: Build Cube with Spark:CBE_DEV[20200102000000_20200103000000] > > Error: > 20/08/08 09:23:54 ERROR ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Error while > instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': > java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Error while > instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:646) > Caused by: java.lang.IllegalArgumentException: Error while instantiating > 'org.apache.spark.sql.hive.HiveSessionStateBuilder': > at > org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1075) > at > org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:142) > at > org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:141) > at scala.Option.getOrElse(Option.scala:121) > at > org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:141) > at > org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:138) > at > org.apache.spark.sql.SparkSession.table(SparkSession.scala:619) > at > org.apache.kylin.engine.spark.SparkUtil.getOtherFormatHiveInput(SparkUtil.java:173) > at > org.apache.kylin.engine.spark.SparkUtil.hiveRecordInputRDD(SparkUtil.java:153) > at > org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:168) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 6 more > Caused by: org.apache.spark.sql.AnalysisException: > java.lang.RuntimeException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can only be issued over connection with kerberos authentication; > at > org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106) > at > org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194) > at > org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105) > at > org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93) > at > org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39) > at > org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54) > at > org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52) > at > org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35) > at > org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289) > at > org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1072) > ... 16 more > Caused by: java.lang.RuntimeException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can only be issued over connection with kerberos authentication > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535) > at > org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:191) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:268) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:362) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266) > at > org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66) > at > org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65) > at > org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:195) > at > org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195) > at > org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:195) > at > org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97) > ... 25 more > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): > Delegation Token can only be issued over connection with kerberos > authentication > at > org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554) > at org.apache.hadoop.ipc.Client.call(Client.java:1498) > at org.apache.hadoop.ipc.Client.call(Client.java:1398) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) > at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:985) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185) > at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source) > at > org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1042) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1689) > at > org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:549) > at > org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:527) > at > org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2400) > at > org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:119) > at > org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:98) > at > org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:200) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:831) > at org.apache.tez.client.TezClient.start(TezClient.java:355) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:184) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532) > ... 39 more > 20/08/08 09:23:54 INFO ApplicationMaster: Final app status: FAILED, exitCode: > 15, (reason: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Error > while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':) > 20/08/08 09:23:54 INFO SparkContext: Invoking stop() from shutdown hook > PS: Job execution working in spark local mode -- This message was sent by Atlassian Jira (v8.3.4#803005)