Gajanan Hebbar created SPARK-30597:
--------------------------------------
Summary: Unable to load properties fine in SparkStandalone HDFS
mode
Key: SPARK-30597
URL: https://issues.apache.org/jira/browse/SPARK-30597
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.3
Environment: 2.4.3
standalone with HDFS
Reporter: Gajanan Hebbar
We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS
mode.
when the application is submitted in standalone HDFS mode the configuration
jar(properties file) is not read when the application is started, and this make
the logger to fall back to default log file and log level.
So when application is submitted to standalone HDFS, configuration files are
not read.
STD OUT LOGS from Standalone HDFS - properties file is not found
==========
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Trying to find [osa-log.properties] using
sun.misc.Launcher$AppClassLoader@4cdf35a9 class loader.
log4j: Trying to find [osa-log.properties] using
ClassLoader.getSystemResource().
log4j: Could not find resource: [osa-log.properties].
log4j: Reading configuration from URL
jar:file:/osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties
STD out from Standalone NFS - properties files are found and able to load it
===================
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Using URL
[jar:file:/scratch/xxxx/gitlocal/soa-osa/out/configdir/ux6m3UQZ/app/sx_DatePipeline_A44C5337_B0D6_4A67_9D60_6BE629DABADA_x0jR7lg5_public/__config__.jar!/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
STD out from YARN HDFS — properties files are found and able to load it
===============
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@2626b418.
log4j: Using URL
[file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
log4j: configuration
file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
