LuciferYang commented on code in PR #46879:
URL: https://github.com/apache/spark/pull/46879#discussion_r1627715299
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:
##########
@@ -1340,6 +1340,15 @@ private[hive] object HiveClientImpl extends Logging {
log"will be reset to 'mr' to disable useless hive logic")
hiveConf.set("hive.execution.engine", "mr", SOURCE_SPARK)
}
+ val cpType = hiveConf.get("datanucleus.connectionPoolingType")
+ // Bonecp might cause memory leak, it could affect some hive client
versions we support
+ // See more details in HIVE-15551
+ // Also, Bonecp is removed in Hive 4.0.0, see HIVE-23258
+ // Here we use DBCP to replace bonecp instead of HikariCP as HikariCP was
introduced in
+ // Hive 2.2.0 (see HIVE-13931) while the minium Hive we support is 2.0.0.
+ if ("bonecp".equalsIgnoreCase(cpType)) {
Review Comment:
Can we switch to using HikariCP in Hive 2.2.0+? Although I haven't tested it
myself, from the official test results of HikariCP, its performance is
significantly better than DBCP:
https://github.com/brettwooldridge/HikariCP-benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
