Onur Satici created SPARK-22479:
-----------------------------------
Summary: SaveIntoDataSourceCommand logs jdbc credentials
Key: SPARK-22479
URL: https://issues.apache.org/jira/browse/SPARK-22479
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Onur Satici
JDBC credentials are not redacted in plans including a
'SaveIntoDataSourceCommand'.
Steps to reproduce:
{code}
spark-shell --packages org.postgresql:postgresql:42.1.1
{code}
{code}
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.util.QueryExecutionListener
val listener = new QueryExecutionListener {
override def onFailure(funcName: String, qe: QueryExecution, exception:
Exception): Unit = {}
override def onSuccess(funcName: String, qe: QueryExecution, duration: Long):
Unit = {
System.out.println(qe.toString())
}
}
spark.listenerManager.register(listener)
spark.range(100).write.format("jdbc").option("url",
"jdbc:postgresql:sparkdb").option("password", "pass").option("driver",
"org.postgresql.Driver").option("dbtable", "test").save()
{code}
The above will yield the following plan:
{code}
== Parsed Logical Plan ==
SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
ErrorIfExists
+- Range (0, 100, step=1, splits=Some(8))
== Analyzed Logical Plan ==
SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
ErrorIfExists
+- Range (0, 100, step=1, splits=Some(8))
== Optimized Logical Plan ==
SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
ErrorIfExists
+- Range (0, 100, step=1, splits=Some(8))
== Physical Plan ==
ExecutedCommand
+- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver ->
org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass),
ErrorIfExists
+- Range (0, 100, step=1, splits=Some(8))
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
