zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1311193343
@xkrogen Thanks for your review. In our cluster, YARN_CONF_DIR is same with
HADOOP_CONF_DIR.
I add some key information about the failed application.
```
# some key information in launch_container.sh:
export
HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/ke/conf/devtest/server/yarn/hadoop"}
export
CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/home/ke/conf/devtest/client-devtest/bigdata/hadoop:/home/ke/bin/hadoop/share/hadoop/common/lib/*:/home/ke/bin/hadoop/share/hadoop/common/*:/home/ke/bin/hadoop/share/hadoop/hdfs:/home/ke/bin/hadoop/share/hadoop/hdfs/lib/*:/home/ke/bin/hadoop/share/hadoop/hdfs/*:/home/ke/bin/hadoop/share/hadoop/mapreduce/lib/*:/home/ke/bin/hadoop/share/hadoop/mapreduce/*:/home/ke/bin/hadoop/share/hadoop/yarn:/home/ke/bin/hadoop/share/hadoop/yarn/lib/*:/home/ke/bin/hadoop/share/hadoop/yarn/*:/home/ke/conf/devtest/client-devtest/bigdata/tez:/home/ke/bin/tez/hadoop-shim-0.9.2.jar:/home
/ke/bin/tez/hadoop-shim-2.7-0.9.2.jar:/home/ke/bin/tez/lib:/home/ke/bin/tez/LICENSE:/home/ke/bin/tez/LICENSE-BSD-3clause:/home/ke/bin/tez/LICENSE-CDDLv1.0:/home/ke/bin/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/home/ke/bin/tez/LICENSE-MIT:/home/ke/bin/tez/LICENSE-SIL_OpenFontLicense-v1.1:/home/ke/bin/tez/NOTICE:/home/ke/bin/tez/tez-api-0.9.2.jar:/home/ke/bin/tez/tez-build-tools-0.9.2.jar:/home/ke/bin/tez/tez-common-0.9.2.jar:/home/ke/bin/tez/tez-dag-0.9.2.jar:/home/ke/bin/tez/tez-examples-0.9.2.jar:/home/ke/bin/tez/tez-ext-service-tests-0.9.2.jar:/home/ke/bin/tez/tez-history-parser-0.9.2.jar:/home/ke/bin/tez/tez-javadoc-tools-0.9.2.jar:/home/ke/bin/tez/tez-job-analyzer-0.9.2.jar:/home/ke/bin/tez/tez-mapreduce-0.9.2.jar:/home/ke/bin/tez/tez-protobuf-history-plugin-0.9.2.jar:/home/ke/bin/tez/tez-runtime-internals-0.9.2.jar:/home/ke/bin/tez/tez-runtime-library-0.9.2.jar:/home/ke/bin/tez/tez-tests-0.9.2.jar:/home/ke/bin/tez/tez-ui-0.9.2.war:/home/ke/bin/tez/tez-yarn-timeline-history-0.9.2.jar:
/home/ke/bin/tez/tez-yarn-timeline-history-with-acls-0.9.2.jar:/home/ke/bin/tez/lib/accessors-smart-1.2.jar:/home/ke/bin/tez/lib/animal-sniffer-annotations-1.17.jar:/home/ke/bin/tez/lib/aopalliance-1.0.jar:/home/ke/bin/tez/lib/asm-5.0.4.jar:/home/ke/bin/tez/lib/async-http-client-1.8.16.jar:/home/ke/bin/tez/lib/audience-annotations-0.5.0.jar:/home/ke/bin/tez/lib/avro-1.7.7.jar:/home/ke/bin/tez/lib/bcpkix-jdk15on-1.60.jar:/home/ke/bin/tez/lib/bcprov-jdk15on-1.60.jar:/home/ke/bin/tez/lib/checker-qual-2.5.2.jar:/home/ke/bin/tez/lib/commons-beanutils-1.9.3.jar:/home/ke/bin/tez/lib/commons-cli-1.2.jar:/home/ke/bin/tez/lib/commons-codec-1.4.jar:/home/ke/bin/tez/lib/commons-collections-3.2.2.jar:/home/ke/bin/tez/lib/commons-collections4-4.1.jar:/home/ke/bin/tez/lib/commons-compress-1.18.jar:/home/ke/bin/tez/lib/commons-configuration2-2.1.1.jar:/home/ke/bin/tez/lib/commons-daemon-1.0.13.jar:/home/ke/bin/tez/lib/commons-io-2.4.jar:/home/ke/bin/tez/lib/commons-lang-2.6.jar:/home/ke/bin/tez/lib
/commons-lang3-3.7.jar:/home/ke/bin/tez/lib/commons-logging-1.1.3.jar:/home/ke/bin/tez/lib/commons-math3-3.1.1.jar:/home/ke/bin/tez/lib/commons-net-3.6.jar:/home/ke/bin/tez/lib/commons-text-1.4.jar:/home/ke/bin/tez/lib/curator-client-2.13.0.jar:/home/ke/bin/tez/lib/curator-framework-2.13.0.jar:/home/ke/bin/tez/lib/curator-recipes-2.13.0.jar:/home/ke/bin/tez/lib/dnsjava-2.1.7.jar:/home/ke/bin/tez/lib/ehcache-3.3.1.jar:/home/ke/bin/tez/lib/error_prone_annotations-2.2.0.jar:/home/ke/bin/tez/lib/failureaccess-1.0.jar:/home/ke/bin/tez/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/ke/bin/tez/lib/gson-2.2.4.jar:/home/ke/bin/tez/lib/guava-27.0-jre.jar:/home/ke/bin/tez/lib/guice-4.0.jar:/home/ke/bin/tez/lib/guice-servlet-4.0.jar:/home/ke/bin/tez/lib/hadoop-annotations-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-auth-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-hdfs-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-hdfs-client-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-map
reduce-client-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-mapreduce-client-core-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-api-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-client-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-registry-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-server-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-server-web-proxy-3.2.1.jar:/home/ke/bin/tez/lib/HikariCP-java7-2.4.12.jar:/home/ke/bin/tez/lib/htrace-core4-4.1.0-incubating.jar:/home/ke/bin/tez/lib/httpclient-4.5.6.jar:/home/ke/bin/tez/lib/httpcore-4.4.10.jar:/home/ke/bin/tez/lib/j2objc-annotations-1.1.jar:/home/ke/bin/tez/lib/jackson-annotations-2.9.8.jar:/home/ke/bin/tez/lib/jackson-core-2.9.8.jar:/home/ke/bin/tez/lib/jackson-core-asl-1.9.2.jar:/home/ke/bin/tez/lib/jackson-databind-2.9.8.jar:/home/ke/bin/tez/lib/jackson-jaxrs-1.9.2.jar:/home/ke/bin/tez/lib/jackson-jaxrs-base-2.9.8.jar:/home/ke/bin/tez/lib/jackson-jaxrs-json-provider-2.9.8.jar:/home/ke/bin
/tez/lib/jackson-mapper-asl-1.9.2.jar:/home/ke/bin/tez/lib/jackson-module-jaxb-annotations-2.9.8.jar:/home/ke/bin/tez/lib/jackson-xc-1.9.2.jar:/home/ke/bin/tez/lib/javax.inject-1.jar:/home/ke/bin/tez/lib/javax.servlet-api-3.1.0.jar:/home/ke/bin/tez/lib/jaxb-api-2.2.11.jar:/home/ke/bin/tez/lib/jaxb-impl-2.2.3-1.jar:/home/ke/bin/tez/lib/jcip-annotations-1.0-1.jar:/home/ke/bin/tez/lib/jersey-client-1.19.jar:/home/ke/bin/tez/lib/jersey-core-1.19.jar:/home/ke/bin/tez/lib/jersey-guice-1.19.jar:/home/ke/bin/tez/lib/jersey-json-1.19.jar:/home/ke/bin/tez/lib/jersey-servlet-1.19.jar:/home/ke/bin/tez/lib/jettison-1.3.4.jar:/home/ke/bin/tez/lib/jetty-http-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-io-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-security-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-server-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-servlet-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-util-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-util-ajax-9.3.24.v2018060
5.jar:/home/ke/bin/tez/lib/jetty-webapp-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-xml-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/json-smart-2.3.jar:/home/ke/bin/tez/lib/jsr305-3.0.0.jar:/home/ke/bin/tez/lib/jsr311-api-1.1.1.jar:/home/ke/bin/tez/lib/kerb-admin-1.0.1.jar:/home/ke/bin/tez/lib/kerb-client-1.0.1.jar:/home/ke/bin/tez/lib/kerb-common-1.0.1.jar:/home/ke/bin/tez/lib/kerb-core-1.0.1.jar:/home/ke/bin/tez/lib/kerb-crypto-1.0.1.jar:/home/ke/bin/tez/lib/kerb-identity-1.0.1.jar:/home/ke/bin/tez/lib/kerb-server-1.0.1.jar:/home/ke/bin/tez/lib/kerb-simplekdc-1.0.1.jar:/home/ke/bin/tez/lib/kerb-util-1.0.1.jar:/home/ke/bin/tez/lib/kerby-asn1-1.0.1.jar:/home/ke/bin/tez/lib/kerby-config-1.0.1.jar:/home/ke/bin/tez/lib/kerby-pkix-1.0.1.jar:/home/ke/bin/tez/lib/kerby-util-1.0.1.jar:/home/ke/bin/tez/lib/kerby-xdr-1.0.1.jar:/home/ke/bin/tez/lib/leveldbjni-all-1.8.jar:/home/ke/bin/tez/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/ke/bin/tez/lib/log4j-1.2.17.jar:
/home/ke/bin/tez/lib/metrics-core-3.1.0.jar:/home/ke/bin/tez/lib/mssql-jdbc-6.2.1.jre7.jar:/home/ke/bin/tez/lib/netty-3.6.2.Final.jar:/home/ke/bin/tez/lib/netty-all-4.0.52.Final.jar:/home/ke/bin/tez/lib/nimbus-jose-jwt-4.41.1.jar:/home/ke/bin/tez/lib/okhttp-2.7.5.jar:/home/ke/bin/tez/lib/okio-1.6.0.jar:/home/ke/bin/tez/lib/paranamer-2.3.jar:/home/ke/bin/tez/lib/protobuf-java-2.5.0.jar:/home/ke/bin/tez/lib/re2j-1.1.jar:/home/ke/bin/tez/lib/RoaringBitmap-0.5.21.jar:/home/ke/bin/tez/lib/servlet-api-2.5.jar:/home/ke/bin/tez/lib/slf4j-api-1.7.10.jar:/home/ke/bin/tez/lib/slf4j-log4j12-1.7.10.jar:/home/ke/bin/tez/lib/snappy-java-1.0.5.jar:/home/ke/bin/tez/lib/stax2-api-3.1.4.jar:/home/ke/bin/tez/lib/stax-api-1.0.1.jar:/home/ke/bin/tez/lib/token-provider-1.0.1.jar:/home/ke/bin/tez/lib/woodstox-core-5.0.3.jar:/home/ke/bin/tez/lib/zookeeper-3.4.13.jar:/home/ke/conf/devtest/client-devtest/bigdata/hive:$PWD/__spark_conf__/__hadoop_conf__"
# some key information in directory.info:
find -L . -maxdepth 5 -ls:
4294973945 0 drwx--s--- 3 bigdata hadoop 159 Nov 11 11:30 .
6442461027 0 drwx--s--- 2 bigdata hadoop 10 Nov 11 11:30 ./tmp
4294973950 24 -rwx------ 1 bigdata hadoop 22397 Nov 11 11:30
./launch_container.sh
4294976100 4 -rw------- 1 bigdata hadoop 511 Nov 11 11:30
./container_tokens
17179897731 0 drwx------ 3 bigdata bigdata 182 Nov 11 11:30
./__spark_conf__
17179897732 4 -r-x------ 1 bigdata bigdata 2083 Nov 11 11:30
./__spark_conf__/log4j.properties
19348229648 0 drwx------ 2 bigdata bigdata 223 Nov 11 11:30
./__spark_conf__/__hadoop_conf__
19348229379 4 -r-x------ 1 bigdata bigdata 3054 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/mapred-site.xml
19348229412 4 -r-x------ 1 bigdata bigdata 702 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/log4j.properties
19348229422 16 -r-x------ 1 bigdata bigdata 14665 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/yarn-site.xml
19348229430 4 -r-x------ 1 bigdata bigdata 3068 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/core-site.xml
19348228517 4 -r-x------ 1 bigdata bigdata 1382 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/switch_hadoop_env.sh
19348228540 16 -r-x------ 1 bigdata bigdata 12710 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/hdfs-site.xml
19348222919 184 -r-x------ 1 bigdata bigdata 184528 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/hive-site.xml
19348222920 8 -r-x------ 1 bigdata bigdata 8052 Nov 11 11:30
./__spark_conf__/__hadoop_conf__/ozone-site.xml
17179897734 208 -r-x------ 1 bigdata bigdata 209545 Nov 11 11:30
./__spark_conf__/__spark_hadoop_conf__.xml
17179897740 4 -r-x------ 1 bigdata bigdata 1138 Nov 11 11:30
./__spark_conf__/__spark_conf__.properties
17179897744 4 -r-x------ 1 bigdata bigdata 617 Nov 11 11:30
./__spark_conf__/__spark_dist_cache__.properties
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
