srowen commented on a change in pull request #24057: [SPARK-26839][WIP][SQL]
Work around classloader changes in Java 9 for Hive isolation
URL: https://github.com/apache/spark/pull/24057#discussion_r264297118
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -214,30 +210,50 @@ private[hive] class IsolatedClientLoader(
private[hive] val classLoader: MutableURLClassLoader = {
val isolatedClassLoader =
if (isolationOn) {
- new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
+ val rootClassLoader: ClassLoader =
+ if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9)) {
+ // In Java 9, the boot classloader can see few JDK classes. The
intended parent
+ // classloader for delegation is now the platform classloader.
+ // See http://java9.wtf/class-loading/
+ val platformCL =
+ classOf[ClassLoader].getMethod("getPlatformClassLoader").
+ invoke(null).asInstanceOf[ClassLoader]
+ // Check to make sure that the root classloader does not know
about Hive.
+
assert(Try(platformCL.loadClass("org.apache.hadoop.hive.conf.HiveConf")).isFailure)
+ platformCL
+ } else {
+ // The boot classloader is represented by null (the instance
itself isn't accessible)
+ // and before Java 9 can see all JDK classes
+ null
}
- def doLoadClass(name: String, resolve: Boolean): Class[_] = {
- val classFileName = name.replaceAll("\\.", "/") + ".class"
- if (isBarrierClass(name)) {
- // For barrier classes, we construct a new copy of the class.
- val bytes =
IOUtils.toByteArray(baseClassLoader.getResourceAsStream(classFileName))
- logDebug(s"custom defining: $name -
${util.Arrays.hashCode(bytes)}")
- defineClass(name, bytes, 0, bytes.length)
- } else if (!isSharedClass(name)) {
- logDebug(s"hive class: $name -
${getResource(classToPath(name))}")
- super.loadClass(name, resolve)
- } else {
- // For shared classes, we delegate to baseClassLoader, but fall
back in case the
- // class is not found.
- logDebug(s"shared class: $name")
- try {
- baseClassLoader.loadClass(name)
- } catch {
- case _: ClassNotFoundException =>
- super.loadClass(name, resolve)
+ if (allJars.isEmpty) {
+ rootClassLoader
Review comment:
This was the tail end of my attempt to get this working; this part doesn't
seem quite right.
In Java 9+, we won't have the JARs from the system classloader to construct
a new ClassLoader here. (There are risky hacks to get it, but not going there
yet.) But this case only comes up in "builtin" mode, where we want to use the
JARs already on the classpath. I figured, why build a new ClassLoader? just use
the current one.
This gets close but now can't find Spark-Hive integration classes like
`HiveClientImpl`.
I think I need to try `baseClassLoader` here, and will do so.
Just raising my current state FYI to @squito , maybe @vanzin or @cloud-fan .
I know it's been years since you've touched it, but maybe @andrewor14 or
@marmbrus wants to see what I'm trying to do here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
