squito commented on issue #23804: [WIP][SPARK-26896] JDK 11 module adjustments for running tests URL: https://github.com/apache/spark/pull/23804#issuecomment-465216199 > I'm unclear why these would only be applicable to test scope? > Is it a problem to just specify a JAXB dependency everywhere rather than make a profile? > > I think we can't work around this by adding a bunch of command line args. Users can't be expected to do this. But I think you're saying it only affects tests. What was the source of the failure? there may be other answers. Yeah, all good questions -- honestly I don't know the answers either, I am not sure this is the right final change, but I thought it might be useful to share as others will hit the same issues. Maybe it is just fine to just add jaxb, even though it'll already be present in jdk8? I'm not sure. The `add-opens` part is much trickier to deal with though. Sure, it would be great to not require the profile & args etc. But you can't pass those args in to java 8, it'll fail immediately because of the unrecognized option. We could change spark code to stop using those internal modules, though that seems like it might be a big task. Or we could try to make spark-submit automatically add those arguments, so users dont' have to worry about them. But we'd still need the maven profile. (though it might be possible to make this maven profile automatically activated if you're using java11, and not require any extra flags.) Regardless, a change like this might be a step forward to get more people building and testing with jdk11?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
