waniuwa opened a new issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx) URL: https://github.com/apache/incubator-dolphinscheduler/issues/2316 In the spark node submission task, configure -- files application.conf in other parameters. Resources also specify files, but the runtime prompts that there is no corresponding application.conf file on HDFS. It works normally to submit spark tasks through the shell. ------------------------------------------------------------------------------ 在spark node提交任务，在其他参数里配置--files application.conf ，资源也指定了文件，但运行时提示hdfs上不存在对应的application.conf 文件。通过shell提交spark任务倒是可以正常运行。 [INFO] 2020-03-26 15:49:47.868 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:47 INFO security.UserGroupInformation: Login successful for user [email protected] using keytab file /opt/keytab/bigdata.keytab [INFO] 2020-03-26 15:49:49.042 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:48 INFO yarn.Client: Attempting to login to the Kerberos using principal: [email protected] and keytab: /opt/keytab/bigdata.keytab 20/03/26 15:49:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm154 20/03/26 15:49:48 INFO yarn.Client: Requesting a new application from cluster with 6 NodeManagers 20/03/26 15:49:48 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (41121 MB per container) 20/03/26 15:49:48 INFO yarn.Client: Will allocate AM container, with 3456 MB memory including 384 MB overhead 20/03/26 15:49:48 INFO yarn.Client: Setting up container launch context for our AM 20/03/26 15:49:48 INFO yarn.Client: Setting up the launch environment for our AM container 20/03/26 15:49:49 INFO yarn.Client: Credentials file set to: credentials-faa7e04b-f09e-40a4-ac42-58ab0f7c9003 [INFO] 2020-03-26 15:49:50.218 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:49 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129 20/03/26 15:49:49 INFO hdfs.DFSClient: Created token for bigdata: HDFS_DELEGATION_TOKEN [email protected], renewer=bigdata, realUser=, issueDate=1585208989034, maxDate=1585813789034, sequenceNumber=117961, masterKeyId=387 on xxxx:8020 20/03/26 15:49:49 INFO yarn.Client: Renewal Interval set to 86400048 20/03/26 15:49:49 INFO yarn.Client: Preparing resources for our AM container 20/03/26 15:49:49 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129 20/03/26 15:49:49 INFO hdfs.DFSClient: Created token for bigdata: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1585208989092, maxDate=1585813789092, sequenceNumber=117962, masterKeyId=387 on xxxx:8020 20/03/26 15:49:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://bigdata0:9083 [INFO] 2020-03-26 15:49:51.223 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1 20/03/26 15:49:50 INFO hive.metastore: Connected to metastore. 20/03/26 15:49:50 INFO hive.metastore: Closed a connection to metastore, current connections: 0 20/03/26 15:49:50 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache. 20/03/26 15:49:50 INFO yarn.Client: Uploading resource file:/opt/keytab/bigdata.keytab -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/bigdata.keytab 20/03/26 15:49:50 INFO yarn.Client: Uploading resource file:/tmp/dolphinscheduler/exec/process/3/7/107/164/offline-analysis-2.jar -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/offline-analysis-2.jar 20/03/26 15:49:51 INFO yarn.Client: Uploading resource file:/tmp/dolphinscheduler/exec/process/3/7/107/164/application.conf -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf 20/03/26 15:49:51 INFO yarn.Client: Uploading resource file:/tmp/spark-4499381f-3954-4240-9edf-cefc3eb58f5a/__spark_conf__2004243660860313056.zip -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/__spark_conf__2004243660860313056.zip 20/03/26 15:49:51 INFO spark.SecurityManager: Changing view acls to: rootadmin,bigdata [INFO] 2020-03-26 15:49:52.478 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:51 INFO spark.SecurityManager: Changing modify acls to: rootadmin,bigdata 20/03/26 15:49:51 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rootadmin, bigdata); users with modify permissions: Set(rootadmin, bigdata) 20/03/26 15:49:51 INFO yarn.Client: Submitting application 129 to ResourceManager 20/03/26 15:49:51 INFO impl.YarnClientImpl: Submitted application application_1585031466521_0129 20/03/26 15:49:52 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:53.484 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:52 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.bigdata queue: root.users.bigdata start time: 1585208991258 final status: UNDEFINED tracking URL: http://bigdata3:8778/proxy/application_1585031466521_0129/ user: bigdata 20/03/26 15:49:53 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:54.486 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:54 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:55.487 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:55 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:56.489 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:56 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:57.490 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:57 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:58.492 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:58 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:49:59.494 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:59 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:50:00.498 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:00 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:50:01.499 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:01 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED) [INFO] 2020-03-26 15:50:02.501 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:02 INFO yarn.Client: Application report for application_1585031466521_0129 (state: FAILED) [INFO] 2020-03-26 15:50:02.908 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:02 INFO yarn.Client: client token: N/A diagnostics: Application application_1585031466521_0129 failed 2 times due to AM Container for appattempt_1585031466521_0129_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://bigdata3:8778/proxy/application_1585031466521_0129/Then, click on links to logs of each attempt. Diagnostics: File does not exist: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf java.io.FileNotFoundException: File does not exist: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1269) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:361) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.bigdata start time: 1585208991258 final status: FAILED tracking URL: http://bigdata3:8778/cluster/app/application_1585031466521_0129 user: bigdata Exception in thread "main" org.apache.spark.SparkException: Application application_1585031466521_0129 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1025) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1072) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 20/03/26 15:50:02 INFO util.ShutdownHookManager: Shutdown hook called 20/03/26 15:50:02 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-4499381f-3954-4240-9edf-cefc3eb58f5a
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
