moomindani commented on a change in pull request #28953:
URL: https://github.com/apache/spark/pull/28953#discussion_r450663945
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
##########
@@ -122,6 +122,37 @@ object JdbcUtils extends Logging {
}
}
+ /**
+ * Runs a custom query against a table from the JDBC database.
+ */
+ def runQuery(conn: Connection, actions: String, options: JDBCOptions): Unit
= {
+ val autoCommit = conn.getAutoCommit
+ conn.setAutoCommit(false)
+ val queries = actions.split(";")
+ try {
+ queries.foreach { query =>
+ val queryString = query.trim()
+ val statement = conn.prepareStatement(queryString)
+ try {
+ statement.setQueryTimeout(options.queryTimeout)
+ val hasResultSet = statement.execute()
Review comment:
Your concern is valid, currently most of JDBC drivers (including MySQL,
PostgreSQL, Oracle, SQL Server, Redshift, Athena, Impala, Snowflake, etc.)
support batchUpdate, but I guess not all drivers support it.
My current implementation followed existing codes in
`JdbcUtils.savePartition()` since it already used `addBatch()` and
`executeBatch()`.
I could come up with following 3 possible directions for it.
(a) Keep current implementation. The `preActions`/`postActions` are only
supported in JDBC drivers which support batchUpdates.
(b) Loop `executeUpdate()` instead of using `addBatch()`/`executeBatch()`.
Although it sacrifices performance benefit, it will be safer when JDBC drivers
do not support batchUpdates.
(c) Add check logic to use `DatabaseMetaData.supportsBatchUpdates()`.
- (c-1). Add it only to `preActions`/`postActions`
- (c-2). For consistency, add it into both `preActions`/`postActions` and
`JdbcUtils.savePartition()`.
Personally I prefer (a) or (b).
If we do (c), we might need to implement two different ways to send queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
