[
https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383709#comment-16383709
]
Thomas Graves commented on SPARK-22479:
---------------------------------------
[~aash] [~onursatici] this seems to have redacted user names as well as the
passwords. We specifically added the User: field to the UI and now its being
blocked, which is makes debugging harder. The user name does not seem like
something that needs to be redacted by default. what is the reasoning behind
that?
Note that at least on yarn there are other ways to easily see the username on
the UI (like the Resource Paths) so its definitely not a complete solution
anyway.
> SaveIntoDataSourceCommand logs jdbc credentials
> -----------------------------------------------
>
> Key: SPARK-22479
> URL: https://issues.apache.org/jira/browse/SPARK-22479
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Onur Satici
> Assignee: Onur Satici
> Priority: Major
> Fix For: 2.2.1, 2.3.0
>
>
> JDBC credentials are not redacted in plans including a
> 'SaveIntoDataSourceCommand'.
> Steps to reproduce:
> {code}
> spark-shell --packages org.postgresql:postgresql:42.1.1
> {code}
> {code}
> import org.apache.spark.sql.execution.QueryExecution
> import org.apache.spark.sql.util.QueryExecutionListener
> val listener = new QueryExecutionListener {
> override def onFailure(funcName: String, qe: QueryExecution, exception:
> Exception): Unit = {}
> override def onSuccess(funcName: String, qe: QueryExecution, duration:
> Long): Unit = {
> System.out.println(qe.toString())
> }
> }
> spark.listenerManager.register(listener)
> spark.range(100).write.format("jdbc").option("url",
> "jdbc:postgresql:sparkdb").option("password", "pass").option("driver",
> "org.postgresql.Driver").option("dbtable", "test").save()
> {code}
> The above will yield the following plan:
> {code}
> == Parsed Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Analyzed Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Optimized Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Physical Plan ==
> ExecutedCommand
> +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
