zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1311254076
@xkrogen Thanks for your review. In our cluster, YARN_CONF_DIR is same
with HADOOP_CONF_DIR.
SparkHadoopUtil.newConfiguration is different from
SparkHadoopUtil.get.newConfiguration
The steps in SparkHadoopUtil.newConfiguration are below:
*(step a) load configuration from config in classpath
*(step b) append s3 and spark.hadoop.* and spark.hive.* configuraiton
The steps in SparkHadoopUtil.get.newConfiguration are below:
*(step a) load configuration from config in classpath
*(step b) append s3 and spark.hadoop.* and spark.hive.* configuraiton
*(step c) append __spark_hadoop_conf__.xml configuration
I add some key information about the failed application.
```
# some key information in launch_container.sh:
export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/XXX/conf/devtest/server/yarn/hadoop"}
export
CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/home/xxx/conf/devtest/client-devtest/bigdata/hadoop:/home/xxx/bin/hadoop/share/hadoop/common/lib/*:/home/xxx/bin/hadoop/share/hadoop/common/*:/home/xxx/bin/hadoop/share/hadoop/hdfs:/home/xxx/bin/hadoop/share/hadoop/hdfs/lib/*:/home/xxx/bin/hadoop/share/hadoop/hdfs/*:/home/xxx/bin/hadoop/share/hadoop/mapreduce/lib/*:/home/xxx/bin/hadoop/share/hadoop/mapreduce/*:/home/xxx/bin/hadoop/share/hadoop/yarn:/home/xxx/bin/hadoop/share/hadoop/yarn/lib/*:/home/xxx/bin/hadoop/share/hadoop/yarn/*:/home/xxx/conf/devtest/client-devtest/bigdata/tez:/home/xxx/bin/tez/hadoop-shim-0.
9.2.jar:/home/xxx/bin/tez/hadoop-shim-2.7-0.9.2.jar:/home/xxx/bin/tez/lib:/home/xxx/bin/tez/LICENSE:/home/xxx/bin/tez/LICENSE-BSD-3clause:/home/xxx/bin/tez/LICENSE-CDDLv1.0:/home/xxx/bin/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/home/xxx/bin/tez/LICENSE-MIT:/home/xxx/bin/tez/LICENSE-SIL_OpenFontLicense-v1.1:/home/xxx/bin/tez/NOTICE:/home/xxx/bin/tez/tez-api-0.9.2.jar:/home/xxx/bin/tez/tez-build-tools-0.9.2.jar:/home/xxx/bin/tez/tez-common-0.9.2.jar:/home/xxx/bin/tez/tez-dag-0.9.2.jar:/home/xxx/bin/tez/tez-examples-0.9.2.jar:/home/xxx/bin/tez/tez-ext-service-tests-0.9.2.jar:/home/xxx/bin/tez/tez-history-parser-0.9.2.jar:/home/xxx/bin/tez/tez-javadoc-tools-0.9.2.jar:/home/xxx/bin/tez/tez-job-analyzer-0.9.2.jar:/home/xxx/bin/tez/tez-mapreduce-0.9.2.jar:/home/xxx/bin/tez/tez-protobuf-history-plugin-0.9.2.jar:/home/xxx/bin/tez/tez-runtime-internals-0.9.2.jar:/home/xxx/bin/tez/tez-runtime-library-0.9.2.jar:/home/xxx/bin/tez/tez-tests-0.9.2.jar:/home/xxx/bin/tez/tez-ui-0.9.2.war:/home/xxx/bin/te
z/tez-yarn-timeline-history-0.9.2.jar:/home/xxx/bin/tez/tez-yarn-timeline-history-with-acls-0.9.2.jar:/home/xxx/bin/tez/lib/accessors-smart-1.2.jar:/home/xxx/bin/tez/lib/animal-sniffer-annotations-1.17.jar:/home/xxx/bin/tez/lib/aopalliance-1.0.jar:/home/xxx/bin/tez/lib/asm-5.0.4.jar:/home/xxx/bin/tez/lib/async-http-client-1.8.16.jar:/home/xxx/bin/tez/lib/audience-annotations-0.5.0.jar:/home/xxx/bin/tez/lib/avro-1.7.7.jar:/home/xxx/bin/tez/lib/bcpkix-jdk15on-1.60.jar:/home/xxx/bin/tez/lib/bcprov-jdk15on-1.60.jar:/home/xxx/bin/tez/lib/checker-qual-2.5.2.jar:/home/xxx/bin/tez/lib/commons-beanutils-1.9.3.jar:/home/xxx/bin/tez/lib/commons-cli-1.2.jar:/home/xxx/bin/tez/lib/commons-codec-1.4.jar:/home/xxx/bin/tez/lib/commons-collections-3.2.2.jar:/home/xxx/bin/tez/lib/commons-collections4-4.1.jar:/home/xxx/bin/tez/lib/commons-compress-1.18.jar:/home/xxx/bin/tez/lib/commons-configuration2-2.1.1.jar:/home/xxx/bin/tez/lib/commons-daemon-1.0.13.jar:/home/xxx/bin/tez/lib/commons-io-2.4.jar:/hom
e/xxx/bin/tez/lib/commons-lang-2.6.jar:/home/xxx/bin/tez/lib/commons-lang3-3.7.jar:/home/xxx/bin/tez/lib/commons-logging-1.1.3.jar:/home/xxx/bin/tez/lib/commons-math3-3.1.1.jar:/home/xxx/bin/tez/lib/commons-net-3.6.jar:/home/xxx/bin/tez/lib/commons-text-1.4.jar:/home/xxx/bin/tez/lib/curator-client-2.13.0.jar:/home/xxx/bin/tez/lib/curator-framework-2.13.0.jar:/home/xxx/bin/tez/lib/curator-recipes-2.13.0.jar:/home/xxx/bin/tez/lib/dnsjava-2.1.7.jar:/home/xxx/bin/tez/lib/ehcache-3.3.1.jar:/home/xxx/bin/tez/lib/error_prone_annotations-2.2.0.jar:/home/xxx/bin/tez/lib/failureaccess-1.0.jar:/home/xxx/bin/tez/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/xxx/bin/tez/lib/gson-2.2.4.jar:/home/xxx/bin/tez/lib/guava-27.0-jre.jar:/home/xxx/bin/tez/lib/guice-4.0.jar:/home/xxx/bin/tez/lib/guice-servlet-4.0.jar:/home/xxx/bin/tez/lib/hadoop-annotations-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-auth-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-hdfs-3.2.1.jar:/
home/xxx/bin/tez/lib/hadoop-hdfs-client-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-mapreduce-client-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-mapreduce-client-core-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-api-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-client-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-registry-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-server-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-server-web-proxy-3.2.1.jar:/home/xxx/bin/tez/lib/HikariCP-java7-2.4.12.jar:/home/xxx/bin/tez/lib/htrace-core4-4.1.0-incubating.jar:/home/xxx/bin/tez/lib/httpclient-4.5.6.jar:/home/xxx/bin/tez/lib/httpcore-4.4.10.jar:/home/xxx/bin/tez/lib/j2objc-annotations-1.1.jar:/home/xxx/bin/tez/lib/jackson-annotations-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-core-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-core-asl-1.9.2.jar:/home/xxx/bin/tez/lib/jackson-databind-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-jaxrs-1.9.2.jar:/home/xxx/bin/tez/lib/
jackson-jaxrs-base-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-jaxrs-json-provider-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-mapper-asl-1.9.2.jar:/home/xxx/bin/tez/lib/jackson-module-jaxb-annotations-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-xc-1.9.2.jar:/home/xxx/bin/tez/lib/javax.inject-1.jar:/home/xxx/bin/tez/lib/javax.servlet-api-3.1.0.jar:/home/xxx/bin/tez/lib/jaxb-api-2.2.11.jar:/home/xxx/bin/tez/lib/jaxb-impl-2.2.3-1.jar:/home/xxx/bin/tez/lib/jcip-annotations-1.0-1.jar:/home/xxx/bin/tez/lib/jersey-client-1.19.jar:/home/xxx/bin/tez/lib/jersey-core-1.19.jar:/home/xxx/bin/tez/lib/jersey-guice-1.19.jar:/home/xxx/bin/tez/lib/jersey-json-1.19.jar:/home/xxx/bin/tez/lib/jersey-servlet-1.19.jar:/home/xxx/bin/tez/lib/jettison-1.3.4.jar:/home/xxx/bin/tez/lib/jetty-http-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-io-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-security-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-server-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-servlet-9.3.24
.v20180605.jar:/home/xxx/bin/tez/lib/jetty-util-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-util-ajax-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-webapp-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-xml-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/json-smart-2.3.jar:/home/xxx/bin/tez/lib/jsr305-3.0.0.jar:/home/xxx/bin/tez/lib/jsr311-api-1.1.1.jar:/home/xxx/bin/tez/lib/kerb-admin-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-client-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-common-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-core-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-crypto-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-identity-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-server-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-simplekdc-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-util-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-asn1-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-config-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-pkix-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-util-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-xdr-1.0.1.jar:/home/xxx/bin/tez/lib/lev
eldbjni-all-1.8.jar:/home/xxx/bin/tez/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/xxx/bin/tez/lib/log4j-1.2.17.jar:/home/xxx/bin/tez/lib/metrics-core-3.1.0.jar:/home/xxx/bin/tez/lib/mssql-jdbc-6.2.1.jre7.jar:/home/xxx/bin/tez/lib/netty-3.6.2.Final.jar:/home/xxx/bin/tez/lib/netty-all-4.0.52.Final.jar:/home/xxx/bin/tez/lib/nimbus-jose-jwt-4.41.1.jar:/home/xxx/bin/tez/lib/okhttp-2.7.5.jar:/home/xxx/bin/tez/lib/okio-1.6.0.jar:/home/xxx/bin/tez/lib/paranamer-2.3.jar:/home/xxx/bin/tez/lib/protobuf-java-2.5.0.jar:/home/xxx/bin/tez/lib/re2j-1.1.jar:/home/xxx/bin/tez/lib/RoaringBitmap-0.5.21.jar:/home/xxx/bin/tez/lib/servlet-api-2.5.jar:/home/xxx/bin/tez/lib/slf4j-api-1.7.10.jar:/home/xxx/bin/tez/lib/slf4j-log4j12-1.7.10.jar:/home/xxx/bin/tez/lib/snappy-java-1.0.5.jar:/home/xxx/bin/tez/lib/stax2-api-3.1.4.jar:/home/xxx/bin/tez/lib/stax-api-1.0.1.jar:/home/xxx/bin/tez/lib/token-provider-1.0.1.jar:/home/xxx/bin/tez/lib/woodstox-core-5.0.3.jar:/home/xxx/bin/tez/lib/
zookeeper-3.4.13.jar:/home/xxx/conf/devtest/client-devtest/bigdata/hive:$PWD/__spark_conf__/__hadoop_conf__"
# some key information in directory.info:
find -L . -maxdepth 5 -ls:
4294973945 0 drwx--s--- 3 bigdata hadoop 159 Nov 11 11:30 .
6442461027 0 drwx--s--- 2 bigdata hadoop 10 Nov 11 11:30 ./tmp
4294973950 24 -rwx------ 1 bigdata hadoop 22397 Nov 11 11:30
./launch_container.sh
4294976100 4 -rw------- 1 bigdata hadoop 511 Nov 11 11:30
./container_tokens
17179897731 0 drwx------ 3 bigdata bigdata 182 Nov 11 11:30
./__spark_conf__
17179897732 4 -r-x------ 1 bigdata bigdata 2083 Nov 11 11:30
./__spark_conf__/log4j.properties
19348229648 0 drwx------ 2 bigdata bigdata 223 Nov 11 11:30
./__spark_conf__/__hadoop_conf__
19348229379 4 -r-x------ 1 bigdata bigdata 3054 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/mapred-site.xml
19348229412 4 -r-x------ 1 bigdata bigdata 702 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/log4j.properties
19348229422 16 -r-x------ 1 bigdata bigdata 14665 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/yarn-site.xml
19348229430 4 -r-x------ 1 bigdata bigdata 3068 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/core-site.xml
19348228517 4 -r-x------ 1 bigdata bigdata 1382 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/switch_hadoop_env.sh
19348228540 16 -r-x------ 1 bigdata bigdata 12710 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/hdfs-site.xml
19348222919 184 -r-x------ 1 bigdata bigdata 184528 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/hive-site.xml
19348222920 8 -r-x------ 1 bigdata bigdata 8052 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/ozone-site.xml
17179897734 208 -r-x------ 1 bigdata bigdata 209545 Nov 11 11:30
./__spark_conf__/__spark_hadoop_conf__.xml
17179897740 4 -r-x------ 1 bigdata bigdata 1138 Nov 11 11:30
./__spark_conf__/__spark_conf__.properties
17179897744 4 -r-x------ 1 bigdata bigdata 617 Nov 11 11:30
./__spark_conf__/__spark_dist_cache__.properties
```
In step a, load config from classpath. Here classpath contains
HADOOP_CONF_DIR and \_\_spark_conf\_\_/\_\_hadoop_conf\_\_. HADOOP_CONF_DIR
happend before \_\_spark_conf\_\_/\_\_hadoop_conf\_\_ , and HADOOP_CONF_DIR is
the conf dir of nodemanager. so
\_\_spark_conf\_\_/\_\_hadoop_conf\_\_/yarn-site.xml will be ignored.
So In this PR, I load \_\_spark_conf\_\_/__spark_hadoop_conf__.xml so that
I can reload config from client.
> Note: It happen in our yarn federation cluster, because the client config
is different from nodemanager client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
