tdg5 opened a new pull request #34745: URL: https://github.com/apache/spark/pull/34745
Augments the `JdbcConnectionProvider` API such that a provider can indicate that it will need to modify the global security configuration when establishing a connection, and as such, access to the global security configuration should be synchronized. Presently, access to the global security configuration is **always** synchronized and this creates a bottleneck for applications that rely on initiating many JDBC connections simultaneously. ### What changes were proposed in this pull request? As suggested by @gaborgsomogyi [here](https://github.com/apache/spark/pull/29024/files#r755788709), augments the `JdbcConnectionProvider` API to include a `needsModifySecurityConfiguration` method that can be used by `ConnectionProvider` to determine when `SecurityConfigurationLock.synchronized` is required to avoid race conditions when establishing a JDBC connection. ### Why are the changes needed? Provides a path forward for working around a significant bottleneck introduced by synchronizing `SecurityConfigurationLock` every time a connection is established. The synchronization isn't always needed and it should be at the discretion of the `JdbcConnectionProvider` to determine when locking is necessary. See [SPARK-37391](https://issues.apache.org/jira/browse/SPARK-37391) or [this thread](https://github.com/apache/spark/pull/29024/files#r754441783). ### Does this PR introduce _any_ user-facing change? Any existing implementations of `JdbcConnectionProvider` will need to add a definition of `needsModifySecurityConfiguration`. I'm also open to adding a default implementation, but it seemed to me that requiring an explicit implementation of the method was preferable. A drop-in implementation that would continue the existing behavior is: ```scala override def needsModifySecurityConfiguration( driver: Driver, options: Map[String, String] ): Boolean = true ``` ### How was this patch tested? Unit tests, but I also plan to run a real workflow once I get the initial thumbs up on this implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
