Note that resources are more properly tied to specific operations and
stages, not to the entire pipeline. This is especially true in the
face of libraries (which should have the ability to declare their own
resources) and cross-language.

On Fri, Nov 8, 2019 at 10:19 AM Łukasz Gajowy <lgaj...@apache.org> wrote:
>
> I figured that it would be good to bump this thread for greater visibility 
> even though I don't have a strong opinion about this (yet - hopefully, I will 
> know more later to be able to share ;) ).
>
> Answering the questions Luke asked will unblock this issue: 
> https://issues.apache.org/jira/browse/BEAM-5495. Solving it is needed for 
> Java 11 migration (current detecting mechanism does not work with java > 8).
>
>
>>
>> That said letting the user resolve the jars to stage can be saner instead of 
>> assuming it is in the classpath/loader. I already have a few cases where it 
>> will fail cause the transforms load the jars from outside the app 
>> classloader (transforms are isolated).
>
>
>
> If I understand correctly, at least in Dataflow runner, if users want to 
> provide custom resources to stage, they can use filesToStage pipeline option. 
> Once the option is not null, the runner doesn't detect the resources 
> automatically and stages resources enlisted in the option instead. I think 
> this should be the approach common for all runners (if it is not the case 
> already).
>
> Thanks,
> Łukasz
>
>
>

Reply via email to