[
https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix K. updated SPARK-22860:
-----------------------------
Description:
The workers log the spark.ssl.keyStorePassword and spark.ssl.trustStorePassword
passed by cli to the executor processes. The ExecutorRunner should escape
passwords to not appear in the worker's log files in INFO level. In this
example, you can see my 'SuperSecretPassword' in a worker log:
{code}
17/12/08 08:04:12 INFO ExecutorRunner: Launch command:
"/global/myapp/oem/jdk/bin/java" "-cp"
"/global/myapp/application/myapp_software/thing_loader_lib/core-repository-model-zzz-1.2.3-SNAPSHOT.jar
[...]
:/global/myapp/application/spark-2.1.1-bin-hadoop2.7/jars/*" "-Xmx16384M"
"-Dspark.authenticate.enableSaslEncryption=true"
"-Dspark.ssl.keyStorePassword=SuperSecretPassword"
"-Dspark.ssl.keyStore=/global/myapp/application/config/ssl/keystore.jks"
"-Dspark.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
"-Dspark.ssl.enabled=true" "-Dspark.driver.port=39927"
"-Dspark.ssl.protocol=TLS" "-Dspark.ssl.trustStorePassword=SuperSecretPassword"
"-Dspark.authenticate=true" "-Dmyapp_IMPORT_DATE=2017-10-30"
"-Dmyapp.config.directory=/global/myapp/application/config"
"-Dsolr.httpclient.builder.factory=com.company.myapp.loader.auth.LoaderConfigSparkSolrBasicAuthConfigurer"
"-Djavax.net.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
"-XX:+UseG1GC" "-XX:+UseStringDeduplication"
"-Dthings.loader.export.zzz_files=false"
"-Dlog4j.configuration=file:/global/myapp/application/config/spark-executor-log4j.properties"
"-XX:+HeapDumpOnOutOfMemoryError" "-XX:+UseStringDeduplication"
"org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
"spark://[email protected]:39927" "--executor-id" "2"
"--hostname" "192.168.0.1" "--cores" "4" "--app-id" "app-20171208080412-0000"
"--worker-url" "spark://[email protected]:59530"
{code}
was:
The workers log the spark.ssl.keyStorePassword and spark.ssl.trustStorePassword
passed by cli to the executor processes. The ExecutorRunner should escape
passwords to not appear in the worker's log files in INFO level. In this
example, you can see my 'SuperSecretPassword' in a worker log:
17/12/08 08:04:12 INFO ExecutorRunner: Launch command:
"/global/myapp/oem/jdk/bin/java" "-cp"
"/global/myapp/application/myapp_software/thing_loader_lib/core-repository-model-zzz-1.2.3-SNAPSHOT.jar
[...]
:/global/myapp/application/spark-2.1.1-bin-hadoop2.7/jars/*" "-Xmx16384M"
"-Dspark.authenticate.enableSaslEncryption=true"
"-Dspark.ssl.keyStorePassword=SuperSecretPassword"
"-Dspark.ssl.keyStore=/global/myapp/application/config/ssl/keystore.jks"
"-Dspark.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
"-Dspark.ssl.enabled=true" "-Dspark.driver.port=39927"
"-Dspark.ssl.protocol=TLS" "-Dspark.ssl.trustStorePassword=SuperSecretPassword"
"-Dspark.authenticate=true" "-Dmyapp_IMPORT_DATE=2017-10-30"
"-Dmyapp.config.directory=/global/myapp/application/config"
"-Dsolr.httpclient.builder.factory=com.company.myapp.loader.auth.LoaderConfigSparkSolrBasicAuthConfigurer"
"-Djavax.net.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
"-XX:+UseG1GC" "-XX:+UseStringDeduplication"
"-Dthings.loader.export.zzz_files=false"
"-Dlog4j.configuration=file:/global/myapp/application/config/spark-executor-log4j.properties"
"-XX:+HeapDumpOnOutOfMemoryError" "-XX:+UseStringDeduplication"
"org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
"spark://[email protected]:39927" "--executor-id" "2"
"--hostname" "192.168.0.1" "--cores" "4" "--app-id" "app-20171208080412-0000"
"--worker-url" "spark://[email protected]:59530"
> Spark workers log ssl passwords passed to the executors
> -------------------------------------------------------
>
> Key: SPARK-22860
> URL: https://issues.apache.org/jira/browse/SPARK-22860
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Felix K.
>
> The workers log the spark.ssl.keyStorePassword and
> spark.ssl.trustStorePassword passed by cli to the executor processes. The
> ExecutorRunner should escape passwords to not appear in the worker's log
> files in INFO level. In this example, you can see my 'SuperSecretPassword' in
> a worker log:
> {code}
> 17/12/08 08:04:12 INFO ExecutorRunner: Launch command:
> "/global/myapp/oem/jdk/bin/java" "-cp"
> "/global/myapp/application/myapp_software/thing_loader_lib/core-repository-model-zzz-1.2.3-SNAPSHOT.jar
> [...]
> :/global/myapp/application/spark-2.1.1-bin-hadoop2.7/jars/*" "-Xmx16384M"
> "-Dspark.authenticate.enableSaslEncryption=true"
> "-Dspark.ssl.keyStorePassword=SuperSecretPassword"
> "-Dspark.ssl.keyStore=/global/myapp/application/config/ssl/keystore.jks"
> "-Dspark.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
> "-Dspark.ssl.enabled=true" "-Dspark.driver.port=39927"
> "-Dspark.ssl.protocol=TLS"
> "-Dspark.ssl.trustStorePassword=SuperSecretPassword"
> "-Dspark.authenticate=true" "-Dmyapp_IMPORT_DATE=2017-10-30"
> "-Dmyapp.config.directory=/global/myapp/application/config"
> "-Dsolr.httpclient.builder.factory=com.company.myapp.loader.auth.LoaderConfigSparkSolrBasicAuthConfigurer"
>
> "-Djavax.net.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks"
> "-XX:+UseG1GC" "-XX:+UseStringDeduplication"
> "-Dthings.loader.export.zzz_files=false"
> "-Dlog4j.configuration=file:/global/myapp/application/config/spark-executor-log4j.properties"
> "-XX:+HeapDumpOnOutOfMemoryError" "-XX:+UseStringDeduplication"
> "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
> "spark://[email protected]:39927" "--executor-id" "2"
> "--hostname" "192.168.0.1" "--cores" "4" "--app-id" "app-20171208080412-0000"
> "--worker-url" "spark://[email protected]:59530"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
