[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-23123. ------------------------------- Resolution: Not A Problem > Unable to run Spark Job with Hadoop NameNode Federation using ViewFS > -------------------------------------------------------------------- > > Key: SPARK-23123 > URL: https://issues.apache.org/jira/browse/SPARK-23123 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 1.6.3 > Reporter: Nihar Nayak > Priority: Major > Labels: Hadoop, Spark > > Added following to core-site.xml in order to make use of ViewFS in a NameNode > federated cluster. > {noformat} > <property> > <name>fs.defaultFS</name> > <value>viewfs:///</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./apps</name> > <value>hdfs://nameservice1/apps</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./app-logs</name> > <value>hdfs://nameservice2/app-logs</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./tmp</name> > <value>hdfs://nameservice2/tmp</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./user</name> > <value>hdfs://nameservice2/user</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./ns1/user</name> > <value>hdfs://nameservice1/user</value> > </property> > <property> > <name>fs.viewfs.mounttable.default.link./ns2/user</name> > <value>hdfs://nameservice2/user</value> > </property> > {noformat} > Got the following error . > {noformat} > spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --num-executors 3 --driver-memory 512m --executor-memory 512m > --executor-cores 1 ${SPARK_HOME}/lib/spark-examples*.jar 10 > 18/01/17 02:14:45 INFO spark.SparkContext: Added JAR > file:/home/nayakxxxx/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar > at spark://xxxxx:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with > timestamp 1516155285534 > 18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over > to rm2 > 18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster > with 26 NodeManagers > 18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not > requested more than the maximum memory capability of the cluster (13800 MB > per container) > 18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB > memory including 384 MB overhead > 18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for > our AM > 18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our > AM container > 18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container > 18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for > namenode: viewfs:/user/nayakxxxx > 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token > 22488202 for nayakxxxx on ha-hdfs:nameservice1 > 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50 > for nayakxxxx on ha-hdfs:nameservice2 > 18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with > URI thrift://XXXX:9083 > 18/01/17 02:14:47 INFO hive.metastore: Connected to metastore. > 18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive > metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61 > 6b 6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e > 52 41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01 > 61 25 f2 42 03 8d 02 21 bb 8e 02 b7 > 18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor > spark.yarn.archive is set, falling back to uploading libraries under > SPARK_HOME. > 18/01/17 02:14:50 INFO yarn.Client: Uploading resource > file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip > -> > viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip > 18/01/17 02:14:55 INFO yarn.Client: Uploading resource > file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip > -> > viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip > 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayakxxxx > 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to: > nayakxxxx > 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls groups to: > 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls groups to: > 18/01/17 02:14:55 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(nayakxxxx); > groups with view permissions: Set(); users with modify permissions: > Set(nayakxxxx); groups with modify permissions: Set() > 18/01/17 02:14:55 INFO yarn.Client: Submitting application > application_1515035441414_275503 to ResourceManager > 18/01/17 02:14:55 INFO impl.YarnClientImpl: Submitted application > application_1515035441414_275503 > 18/01/17 02:14:55 INFO cluster.SchedulerExtensionServices: Starting Yarn > extension services with app application_1515035441414_275503 and attemptId > None > 18/01/17 02:14:56 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:14:56 INFO yarn.Client: > client token: Token { kind: YARN_CLIENT_TOKEN, service: } > diagnostics: AM container is launched, waiting for AM container to > Register with RM > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: common > start time: 1516155295100 > final status: UNDEFINED > tracking URL: > http://XXXX:8088/proxy/application_1515035441414_275503/ > user: nayakxxxx > 18/01/17 02:14:57 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:14:58 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:14:59 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:00 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:01 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:02 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:03 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:04 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:05 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:06 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:07 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:08 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:09 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:10 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:11 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:12 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:13 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:14 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:15 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:16 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:17 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:18 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:19 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:20 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:21 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:22 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:23 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:24 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:25 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:26 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:27 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:28 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:29 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:30 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:31 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:32 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:33 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: ACCEPTED) > 18/01/17 02:15:34 INFO yarn.Client: Application report for > application_1515035441414_275503 (state: FAILED) > 18/01/17 02:15:34 INFO yarn.Client: > client token: N/A > diagnostics: Application application_1515035441414_275503 failed 10 > times due to AM Container for appattempt_1515035441414_275503_000010 exited > with exitCode: -1000 > For more detailed output, check the application tracking page: > http://xxxxx:8088/cluster/app/application_1515035441414_275503 Then click on > links to logs of each attempt. > Diagnostics: ViewFs: Cannot initialize: Empty Mount table in config for > viewfs://default/ > java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config > for viewfs://default/ > at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:168) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:168) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) > at > org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358) > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Failing this attempt. Failing the application. > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: common > start time: 1516155295100 > final status: FAILED > tracking URL: > http://xxxxxx:8088/cluster/app/application_1515035441414_275503 > user: nayakxxxx > 18/01/17 02:15:34 INFO yarn.Client: Deleted staging directory > viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503 > 18/01/17 02:15:34 ERROR spark.SparkContext: Error initializing SparkContext. > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org