holdenk commented on issue #221: URL: https://github.com/apache/arrow-datafusion-comet/issues/221#issuecomment-2014296296
> > You can work around this error by instead of using --jars to add the arrow datafusion comet jar to the classpath instead copying it into Spark's jar directory so it will be loaded with the same classloader. > > > > Emmm, it could be a potential solution. But it seems a bit of inconvenient. Per my understanding, it usually requires extra effort to change Spark's jar directory/archive in the production environment. > True, especially for users of a vendor solution, although for my deployments this isn't a big deal (we package our own Spark version anyways). Let me take another look next week and see if there is a way to get loaded with Sparks default class loader. > > > > since changing the Spark class loader is not easy (I also tried with user classpath first class loader) > > > > So this issue occurred regardless the `spark.driver.userClassPathFirst` setting being true or false? Yup :( Now I only tried in vanilla 3.4. > > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
