Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10519#discussion_r48762926
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -34,10 +35,31 @@ import org.apache.spark.sql.{DataFrame, Row}
object JdbcUtils extends Logging {
/**
- * Establishes a JDBC connection.
+ * Returns a factory for creating connections to the given JDBC URL.
+ *
+ * @param url the JDBC url to connect to.
+ * @param properties JDBC connection properties.
*/
- def createConnection(url: String, connectionProperties: Properties):
Connection = {
- JDBCRDD.getConnector(connectionProperties.getProperty("driver"), url,
connectionProperties)()
+ def createConnectionFactory(url: String, properties: Properties): () =>
Connection = {
+ val userSpecifiedDriverClass = Option(properties.getProperty("driver"))
+ userSpecifiedDriverClass.foreach(DriverRegistry.register)
+ // Performing this part of the logic on the driver guards against the
corner-case where the
+ // driver returned for a URL is different on the driver and executors
due to classpath
+ // differences.
+ val driverClass: String = userSpecifiedDriverClass.getOrElse {
+ DriverManager.getDriver(url).getClass.getCanonicalName
+ }
+ () => {
+ userSpecifiedDriverClass.foreach(DriverRegistry.register)
+ val driver: Driver = DriverManager.getDrivers.asScala.collectFirst {
+ case d: DriverWrapper if d.wrapped.getClass.getCanonicalName ==
driverClass => d
--- End diff --
This is the only real bit of trickiness here and was the part that was
missing from my `spark-redshift` patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
