I believe the closest suggestion[1] we had that worked for Java 11 and maintained backwards compatibility was to use the URLClassLoader to infer the resources and if we couldn't do that then look at the java.class.path system property to do the inference otherwise fail and force the users to tell us what. There are too many scenarios where we will do it wrong because of how people package and deploy their code whether it is an embedded application server or some other application container with a security manager that will prevent us from doing the right thing.
On Fri, Nov 8, 2019 at 10:31 AM Robert Bradshaw <rober...@google.com> wrote: > Note that resources are more properly tied to specific operations and > stages, not to the entire pipeline. This is especially true in the > face of libraries (which should have the ability to declare their own > resources) and cross-language. > > On Fri, Nov 8, 2019 at 10:19 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > > > > I figured that it would be good to bump this thread for greater > visibility even though I don't have a strong opinion about this (yet - > hopefully, I will know more later to be able to share ;) ). > > > > Answering the questions Luke asked will unblock this issue: > https://issues.apache.org/jira/browse/BEAM-5495. Solving it is needed for > Java 11 migration (current detecting mechanism does not work with java > 8). > > > > > >> > >> That said letting the user resolve the jars to stage can be saner > instead of assuming it is in the classpath/loader. I already have a few > cases where it will fail cause the transforms load the jars from outside > the app classloader (transforms are isolated). > > > > > > > > If I understand correctly, at least in Dataflow runner, if users want to > provide custom resources to stage, they can use filesToStage pipeline > option. Once the option is not null, the runner doesn't detect the > resources automatically and stages resources enlisted in the option > instead. I think this should be the approach common for all runners (if it > is not the case already). > Your understanding is correct and consistency across runners for a pipeline option is good for our users. > > > > Thanks, > > Łukasz > > > > > > > 1: https://github.com/apache/beam/pull/8775
