xkrogen commented on PR #40144: URL: https://github.com/apache/spark/pull/40144#issuecomment-1444004552
Great question @cloud-fan , and actually no, we don't. For all of the other values of `spark.sql.hive.metastore.jars` besides 'builtin', the user JARs are not included at all ([refer to this section of HiveUtils](https://github.com/apache/spark/blob/87e3d5625e76bb734b8dd753bfb25002822c8585/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala#L451-L513)). In all of those cases, the JAR list is constructed purely from the dependencies for the specified Hive version. Whether that behavior is correct is another question -- @shardulm94 informed me that user JARs are required to support custom serdes inside of the Hive client -- but in any case, 'builtin' is the only mode that is susceptible to this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
