Re: Detecting resources to stage

2019-11-27 Thread Gleb Kanterov
Agree, this makes sense. On Wed, Nov 27, 2019 at 6:23 PM Luke Cwik wrote: > That looks good as well. > > I would suggest that we make the classpath scanning system pluggable using > PipelineOptions. For example in GcpOptions[1], we use two default instance > factories. The first one controls

Re: Detecting resources to stage

2019-11-27 Thread Luke Cwik
That looks good as well. I would suggest that we make the classpath scanning system pluggable using PipelineOptions. For example in GcpOptions[1], we use two default instance factories. The first one controls which class is used as the factory[2] and the second one instantiates an instance of

Re: Detecting resources to stage

2019-11-27 Thread Gleb Kanterov
I didn't think it through, but this is something I have in mind. Keep existing implementation for URLClassLoader, and use URLClassLoader for experimental support of Java 11. List urls; if (classLoader instanceof URLClassLoader) { urls = Arrays.asList(((URLClassLoader)

Re: Detecting resources to stage

2019-11-27 Thread Łukasz Gajowy
This looks promising. Do you think you could share your code as well? That part sounds very calming: "ClassGraph is fully compatible with the new JPMS module system (Project Jigsaw / JDK 9+), i.e. it can scan both the traditional classpath and the module path. However, the code is also fully

Re: Detecting resources to stage

2019-11-27 Thread Gleb Kanterov
Today I tried using classgraph [1] library to scan classpath in Java 11 instead of using URLClassLoader, and after that, the job worked on Dataflow. The logic of scanning classpath is pretty sophisticated [2], and classgraph doesn't have any dependencies. I'm wondering if we can relocate it to

Re: Detecting resources to stage

2019-11-08 Thread Luke Cwik
I believe the closest suggestion[1] we had that worked for Java 11 and maintained backwards compatibility was to use the URLClassLoader to infer the resources and if we couldn't do that then look at the java.class.path system property to do the inference otherwise fail and force the users to tell

Re: Detecting resources to stage

2019-11-08 Thread Robert Bradshaw
Note that resources are more properly tied to specific operations and stages, not to the entire pipeline. This is especially true in the face of libraries (which should have the ability to declare their own resources) and cross-language. On Fri, Nov 8, 2019 at 10:19 AM Łukasz Gajowy wrote: > > I

Re: Detecting resources to stage

2019-11-08 Thread Łukasz Gajowy
I figured that it would be good to bump this thread for greater visibility even though I don't have a strong opinion about this (yet - hopefully, I will know more later to be able to share ;) ). Answering the questions Luke asked will unblock this issue: