Today I tried using classgraph [1] library to scan classpath in Java 11 instead of using URLClassLoader, and after that, the job worked on Dataflow. The logic of scanning classpath is pretty sophisticated [2], and classgraph doesn't have any dependencies. I'm wondering if we can relocate it to java-core jar and use in for non-URLClassLoaders?
[1]: https://github.com/classgraph/classgraph [2]: https://github.com/classgraph/classgraph/blob/master/src/main/java/io/github/classgraph/Scanner.java On Fri, Nov 8, 2019 at 11:40 PM Luke Cwik <lc...@google.com> wrote: > I believe the closest suggestion[1] we had that worked for Java 11 and > maintained backwards compatibility was to use the URLClassLoader to infer > the resources and if we couldn't do that then look at the java.class.path > system property to do the inference otherwise fail and force the users to > tell us what. There are too many scenarios where we will do it wrong > because of how people package and deploy their code whether it is an > embedded application server or some other application container with a > security manager that will prevent us from doing the right thing. > > On Fri, Nov 8, 2019 at 10:31 AM Robert Bradshaw <rober...@google.com> > wrote: > >> Note that resources are more properly tied to specific operations and >> stages, not to the entire pipeline. This is especially true in the >> face of libraries (which should have the ability to declare their own >> resources) and cross-language. >> >> On Fri, Nov 8, 2019 at 10:19 AM Łukasz Gajowy <lgaj...@apache.org> wrote: >> > >> > I figured that it would be good to bump this thread for greater >> visibility even though I don't have a strong opinion about this (yet - >> hopefully, I will know more later to be able to share ;) ). >> > >> > Answering the questions Luke asked will unblock this issue: >> https://issues.apache.org/jira/browse/BEAM-5495. Solving it is needed >> for Java 11 migration (current detecting mechanism does not work with java >> > 8). >> > >> > >> >> >> >> That said letting the user resolve the jars to stage can be saner >> instead of assuming it is in the classpath/loader. I already have a few >> cases where it will fail cause the transforms load the jars from outside >> the app classloader (transforms are isolated). >> > >> > >> > >> > If I understand correctly, at least in Dataflow runner, if users want >> to provide custom resources to stage, they can use filesToStage pipeline >> option. Once the option is not null, the runner doesn't detect the >> resources automatically and stages resources enlisted in the option >> instead. I think this should be the approach common for all runners (if it >> is not the case already). >> > > Your understanding is correct and consistency across runners for a > pipeline option is good for our users. > > >> > >> > Thanks, >> > Łukasz >> > >> > >> > >> > > 1: https://github.com/apache/beam/pull/8775 >
