[
https://issues.apache.org/jira/browse/SPARK-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246328#comment-16246328
]
Andrew Ash commented on SPARK-22479:
------------------------------------
Completely agree that credentials shouldn't be in the toString since query
plans are logged in many places. This looks like it brings
SaveIntoDataSourceCommand more in-line with JdbcRelation, which also currently
redacts credentials from its toString to avoid them being written to logs.
> SaveIntoDataSourceCommand logs jdbc credentials
> -----------------------------------------------
>
> Key: SPARK-22479
> URL: https://issues.apache.org/jira/browse/SPARK-22479
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Onur Satici
>
> JDBC credentials are not redacted in plans including a
> 'SaveIntoDataSourceCommand'.
> Steps to reproduce:
> {code}
> spark-shell --packages org.postgresql:postgresql:42.1.1
> {code}
> {code}
> import org.apache.spark.sql.execution.QueryExecution
> import org.apache.spark.sql.util.QueryExecutionListener
> val listener = new QueryExecutionListener {
> override def onFailure(funcName: String, qe: QueryExecution, exception:
> Exception): Unit = {}
> override def onSuccess(funcName: String, qe: QueryExecution, duration:
> Long): Unit = {
> System.out.println(qe.toString())
> }
> }
> spark.listenerManager.register(listener)
> spark.range(100).write.format("jdbc").option("url",
> "jdbc:postgresql:sparkdb").option("password", "pass").option("driver",
> "org.postgresql.Driver").option("dbtable", "test").save()
> {code}
> The above will yield the following plan:
> {code}
> == Parsed Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Analyzed Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Optimized Logical Plan ==
> SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> == Physical Plan ==
> ExecutedCommand
> +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
> ErrorIfExists
> +- Range (0, 100, step=1, splits=Some(8))
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
