[
https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881773#comment-15881773
]
Apache Spark commented on SPARK-19720:
--------------------------------------
User 'markgrover' has created a pull request for this issue:
https://github.com/apache/spark/pull/17047
> Redact sensitive information from SparkSubmit console output
> ------------------------------------------------------------
>
> Key: SPARK-19720
> URL: https://issues.apache.org/jira/browse/SPARK-19720
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.2.0
> Reporter: Mark Grover
>
> SPARK-18535 took care of redacting sensitive information from Spark event
> logs and UI. However, it intentionally didn't bother redacting the same
> sensitive information from SparkSubmit's console output because it was on the
> client's machine, which already had the sensitive information on disk (in
> spark-defaults.conf) or on terminal (spark-submit command line).
> However, it seems now that it's better to redact information from
> SparkSubmit's console output as well because orchestration software like
> Oozie usually expose SparkSubmit's console output via a UI. To make matters
> worse, Oozie, in particular, always sets the {{--verbose}} flag on
> SparkSubmit invocation, making the sensitive information readily available in
> its UI (see
> [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248]
> here).
> This is a JIRA for tracking redaction of sensitive information from
> SparkSubmit's console output.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
