Craiu Constantin-Tiberiu created SPARK-60507:
------------------------------------------------
Summary: Reuse schema resolution connection to read JDBC database
metadata
Key: SPARK-60507
URL: https://issues.apache.org/jira/browse/SPARK-60507
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Craiu Constantin-Tiberiu
{{JDBCRDD.scanTable}} reads metadata through
{{{}JDBCDatabaseMetadata.fromJDBCConnectionFactory{}}}, which opens a new
driver-side connection on every call.
For a V1 JDBC read ({{{}spark.read.jdbc{}}}, {{{}format("jdbc").load(){}}}),
schema resolution already opens one connection so we can just reuse that one
isntead of opening a new connection to the DB.
This is wasteful for sources where connecting is expensive (TLS, auth
handshakes, connection limits), and it is unnecessary because the metadata can
be read on the connection used to resolve the schema.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
