Zouxxyy commented on issue #6543: URL: https://github.com/apache/hudi/issues/6543#issuecomment-1236082037
Maybe I can answer your question. First, hudi uses `ReflectionUtils.java` 's `Class.forName(clazzName)` to load class. The classloader it used is determined by `ReflectionUtils`, which is determined by `hudi-spark-bundle.jar`. Then, spark has its own defined class loaders, `MutableURLClassLoader` and `ParentClassLoader`, which recognize classes defined by sparkconf `--jars` or `spark.jars` Besides, java's `AppClassLoader` can recognize classes in java classpath, which can be defined by sparkconf `spark.driver.extraClassPath` According to java's class loading mechanism, `MutableURLClassLoader` can see all classes loaded by `AppClassLoader`, but `AppClassLoader` can not see any class loaded by `MutableURLClassLoader` If your `hudi-spark-bundle.jar` is s defined by `spark.driver.extraClassPath` and your customer jar is defined by `--jars` or `spark.jars`, `ReflectionUtils` will use `AppClassLoader` to load your `customer jar`, since it is not defined In the java classpath, there will be `java.lang.ClassnotFoundException`. I guess that's what you probably did **So I think configing your `customer.jar` in `spark.driver.extraClassPath` will solve your problem** Finally, why you were able to succeed before, I guess because of this patch [HUDI-1750](https://github.com/apache/hudi/pull/2753), which was modified back in 0.10.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
