[jira] [Created] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS
Nihar Nayak created SPARK-23123:
---
Summary: Unable to run Spark Job with Hadoop NameNode Federation
using ViewFS
Key: SPARK-23123
URL: https://issues.apache.org/jira/browse/SPARK-23123
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 1.6.3
Reporter: Nihar Nayak
Added following to core-site.xml in order to make use of ViewFS in a NameNode
federated cluster.
{noformat}
fs.defaultFS
viewfs:///
fs.viewfs.mounttable.default.link./apps
hdfs://nameservice1/apps
fs.viewfs.mounttable.default.link./app-logs
hdfs://nameservice2/app-logs
fs.viewfs.mounttable.default.link./tmp
hdfs://nameservice2/tmp
fs.viewfs.mounttable.default.link./user
hdfs://nameservice2/user
fs.viewfs.mounttable.default.link./ns1/user
hdfs://nameservice1/user
fs.viewfs.mounttable.default.link./ns2/user
hdfs://nameservice2/user
{noformat}
Got the following error .
{noformat}
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client
--num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores
1 ${SPARK_HOME}/lib/spark-examples*.jar 10
18/01/17 02:14:45 INFO spark.SparkContext: Added JAR
file:/home/nayak/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar
at spark://x:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with
timestamp 1516155285534
18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster
with 26 NodeManagers
18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (13800 MB per container)
18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for our
AM
18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our
AM container
18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container
18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for
namenode: viewfs:/user/nayak
18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
22488202 for nayak on ha-hdfs:nameservice1
18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50
for nayak on ha-hdfs:nameservice2
18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with URI
thrift://:9083
18/01/17 02:14:47 INFO hive.metastore: Connected to metastore.
18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive
metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61 6b
6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e 52
41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01 61
25 f2 42 03 8d 02 21 bb 8e 02 b7
18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor
spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/01/17 02:14:50 INFO yarn.Client: Uploading resource
file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip
->
viewfs:/user/nayak/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip
18/01/17 02:14:55 INFO yarn.Client: Uploading resource
file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip
->
viewfs:/user/nayak/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip
18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayak
18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to: nayak
18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls groups to:
18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls groups to:
18/01/17 02:14:55 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(nayak);
groups with view permissions: Set(); users with modify permissions:
Set(nayak); groups with modify permissions: Set()
18/01/17 02:14:55 INFO yarn.Client: Submitting application
application_1515035441414_275503 to ResourceManager
18/01/17 02:14:55 INFO impl.YarnClientImpl: Submitted application
application_1515035441414_275503
18/01/17 02:14:55 INFO cluster.SchedulerExtensionServices: Starting Yarn
extension services with app application_1515035441414_275503 and attemptId None
18/01/17 02:14:56 INFO yarn.Client: Application report for
application_1515035441414_275503 (state: ACCEPTED)
18/01/17 02:14:56 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to
Register with RM
ApplicationMa
[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS
[
https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328349#comment-16328349
]
Nihar Nayak commented on SPARK-23123:
-
I'm able to run all other Hadoop Applications (MR/Hive Jobs, HDFS Command as
well). It's only failing in case of Spark application. As far as the exception
is concerned, yes it's thrown by Hadoop, but the exception saying unable to
initialize which is strange because all the related configurations are provided
in core-site.xml .
Note : I can able reproduce the exact issue in case of HDFS command if i remove
the "fs.viewfs.mounttable" configuration from core-site.xml, but with the
correct configuration any Hadoop dependent application (Including Spark) should
run without any issue .
> Unable to run Spark Job with Hadoop NameNode Federation using ViewFS
>
>
> Key: SPARK-23123
> URL: https://issues.apache.org/jira/browse/SPARK-23123
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
>Affects Versions: 1.6.3
>Reporter: Nihar Nayak
>Priority: Major
> Labels: Hadoop, Spark
>
> Added following to core-site.xml in order to make use of ViewFS in a NameNode
> federated cluster.
> {noformat}
>
> fs.defaultFS
> viewfs:///
>
>
> fs.viewfs.mounttable.default.link./apps
> hdfs://nameservice1/apps
>
>
> fs.viewfs.mounttable.default.link./app-logs
> hdfs://nameservice2/app-logs
>
>
> fs.viewfs.mounttable.default.link./tmp
> hdfs://nameservice2/tmp
>
>
> fs.viewfs.mounttable.default.link./user
> hdfs://nameservice2/user
>
>
> fs.viewfs.mounttable.default.link./ns1/user
> hdfs://nameservice1/user
>
>
> fs.viewfs.mounttable.default.link./ns2/user
> hdfs://nameservice2/user
>
> {noformat}
> Got the following error .
> {noformat}
> spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client
> --num-executors 3 --driver-memory 512m --executor-memory 512m
> --executor-cores 1 ${SPARK_HOME}/lib/spark-examples*.jar 10
> 18/01/17 02:14:45 INFO spark.SparkContext: Added JAR
> file:/home/nayak/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar
> at spark://x:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with
> timestamp 1516155285534
> 18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm2
> 18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster
> with 26 NodeManagers
> 18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not
> requested more than the maximum memory capability of the cluster (13800 MB
> per container)
> 18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB
> memory including 384 MB overhead
> 18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for
> our AM
> 18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our
> AM container
> 18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container
> 18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for
> namenode: viewfs:/user/nayak
> 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 22488202 for nayak on ha-hdfs:nameservice1
> 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50
> for nayak on ha-hdfs:nameservice2
> 18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://:9083
> 18/01/17 02:14:47 INFO hive.metastore: Connected to metastore.
> 18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive
> metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61
> 6b 6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e
> 52 41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01
> 61 25 f2 42 03 8d 02 21 bb 8e 02 b7
> 18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor
> spark.yarn.archive is set, falling back to uploading libraries under
> SPARK_HOME.
> 18/01/17 02:14:50 INFO yarn.Client: Uploading resource
> file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip
> ->
> viewfs:/user/nayak/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip
> 18/01/17 02:14:55 INFO yarn.Client: Uploading resource
> file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip
> ->
> viewfs:/user/nayak/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayak
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to:
> nayak
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changin
