gaborgsomogyi commented on a change in pull request #29964:
URL: https://github.com/apache/spark/pull/29964#discussion_r501578769
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -22,12 +22,15 @@ import java.util.ServiceLoader
import scala.collection.mutable
+import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.internal.Logging
import org.apache.spark.security.SecurityConfigurationLock
import org.apache.spark.sql.jdbc.JdbcConnectionProvider
import org.apache.spark.util.Utils
private[jdbc] object ConnectionProvider extends Logging {
+ private val providerEnabledConfig =
"spark.security.jdbc.connection.provider.%s.enabled"
Review comment:
It makes sense to make it SQL like so adapting. What I think however we
need blacklist and not whitelist. Having a blacklist doesn't expect any
configuration from user when new provider added.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -46,8 +49,13 @@ private[jdbc] object ConnectionProvider extends Logging {
logError(s"Failed to load built in provider.", t)
}
}
- // Seems duplicate but it's needed for Scala 2.13
- providers.toSeq
+
+ // toSeq seems duplicate but it's needed for Scala 2.13
+ val sparkConf = SparkEnv.get.conf
+ providers.filter { p =>
+ val key = providerEnabledConfig.format(p.name)
+ sparkConf.getOption(key).forall(_.toBoolean)
Review comment:
Adding more descriptive message.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -22,12 +22,15 @@ import java.util.ServiceLoader
import scala.collection.mutable
+import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.internal.Logging
import org.apache.spark.security.SecurityConfigurationLock
import org.apache.spark.sql.jdbc.JdbcConnectionProvider
import org.apache.spark.util.Utils
private[jdbc] object ConnectionProvider extends Logging {
+ private val providerEnabledConfig =
"spark.security.jdbc.connection.provider.%s.enabled"
Review comment:
Just changed it.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -46,8 +49,13 @@ private[jdbc] object ConnectionProvider extends Logging {
logError(s"Failed to load built in provider.", t)
}
}
- // Seems duplicate but it's needed for Scala 2.13
- providers.toSeq
+
+ // toSeq seems duplicate but it's needed for Scala 2.13
Review comment:
Fixed.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -46,8 +49,13 @@ private[jdbc] object ConnectionProvider extends Logging {
logError(s"Failed to load built in provider.", t)
}
}
- // Seems duplicate but it's needed for Scala 2.13
- providers.toSeq
+
+ // toSeq seems duplicate but it's needed for Scala 2.13
+ val sparkConf = SparkEnv.get.conf
+ providers.filter { p =>
+ val key = providerEnabledConfig.format(p.name)
+ sparkConf.getOption(key).forall(_.toBoolean)
Review comment:
Code changed so not needed.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/DB2ConnectionProvider.scala
##########
@@ -28,6 +28,8 @@ import
org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions
private[sql] class DB2ConnectionProvider extends SecureConnectionProvider {
override val driverClass = "com.ibm.db2.jcc.DB2Driver"
+ override def name: String = "db2"
Review comment:
Fixed.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##########
@@ -46,8 +47,10 @@ private[jdbc] object ConnectionProvider extends Logging {
logError(s"Failed to load built in provider.", t)
}
}
- // Seems duplicate but it's needed for Scala 2.13
- providers.toSeq
+
+ val disabledProviders =
SQLConf.get.disabledJdbcConnectionProviders.split(",")
Review comment:
Changed, however I've taken this solution from existing params. After
this I'm going to file a PR to change them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
