This is great, let me try it out. Best, Ke
> On Sep 30, 2021, at 6:06 PM, Robert Bradshaw <rober...@google.com> wrote: > > On Thu, Sep 30, 2021 at 6:00 PM Ke Wu <ke.wu...@gmail.com> wrote: >> >> I am able to annotate/mark a java transform by setting its resource hints >> [1] as well, which resulted in a different environment id, e.g. >> >> beam:env:docker:v1 VS beam:env:docker:v11 >> >> Is this on the right track? > > Yep. > >> If Yes, I suppose then I need to configure job bundle factory to be able to >> understand multiple environments and configure them separately as well. > > It should already do the right thing here. That's how multi-language works. > >> [1] >> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java#L218 >> >> On Sep 30, 2021, at 10:34 AM, Robert Bradshaw <rober...@google.com> wrote: >> >> On Thu, Sep 30, 2021 at 9:25 AM Ke Wu <ke.wu...@gmail.com> wrote: >> >> >> Ideally, we do not want to expose anything directly to users and we, as the >> framework and platform provider, separate things out under the hood. >> >> I would expect users to author their DoFn(s) in the same way as they do >> right now, but we expect to change the DoFn(s) that we provide, will be >> annotated/marked so that it can be recognized during runtime. >> >> In our use case, application is executed in Kubernetes environment >> therefore, we are expecting to directly use different docker image to >> isolate dependencies. >> >> e.g. we have docker image A, which is beam core, that is used to start job >> server and runner process. We have a docker image B, which contains DoFn(s) >> that platform provides to serve as a external worker pool service to execute >> platform provided DoFn(s), last but not least, users would have their own >> docker image represent their application, which will be used to start the >> external worker pool service to handle their own UDF execution. >> >> Does this make sense ? >> >> >> In Python it's pretty trivial to annotate transforms (e.g. the >> "platform" transforms) which could be used to mark their environments >> prior to optimization (e.g. fusion). As mentioned, you could use >> resource hints (even a "dummy" hint like >> "use_platform_environment=True") to force these into a separate docker >> image as well. >> >> On Sep 29, 2021, at 1:09 PM, Luke Cwik <lc...@google.com> wrote: >> >> That sounds neat. I think that before you try to figure out how to change >> Beam to fit this usecase is to think about what would be the best way for >> users to specify these requirements when they are constructing the pipeline. >> Once you have some samples that you could share the community would probably >> be able to give you more pointed advice. >> For example will they be running one application with a complicated class >> loader setup, if so then we could probably do away with multiple >> environments and try to have DoFn's recognize their specific class loader >> configuration and replicate it on the SDK harness side. >> >> Also, for performance reasons users may want to resolve their dependency >> issues to create a maximally fused graph to limit performance impact due to >> the encoding/decoding boundaries at the edges of those fused graphs. >> >> Finally, this could definitely apply to languages like Python and Go (now >> that Go has support for modules) as dependency issues are a common problem. >> >> >> On Wed, Sep 29, 2021 at 11:47 AM Ke Wu <ke.wu...@gmail.com> wrote: >> >> >> Thanks for the advice. >> >> Here are some more background: >> >> We are building a feature called “split deployment” such that, we can >> isolate framework/platform core from user code/dependencies to address >> couple of operational challenges such as dependency conflict, >> alert/exception triaging. >> >> With Beam’s portability framework, runner and sdk worker process naturally >> decouples beam core and user UDFs(DoFn), which is awesome! On top of this, >> we could further distinguish DoFn(s) that end user authors from DoFn(s) that >> platform provides, therefore, we would like these DoFn(s) to be executed in >> different environments, even in the same language, e.g. Java. >> >> Therefore, I am exploring approaches and recommendations what are the proper >> way to do that. >> >> Let me know your thoughts, any feedback/advice is welcome. >> >> Best, >> Ke >> >> On Sep 27, 2021, at 11:56 AM, Luke Cwik <lc...@google.com> wrote: >> >> Resource hints have a limited use case and might fit your need. >> You could also try to use the expansion service XLang route to bring in a >> different Java environment. >> Finally, you could modify the pipeline proto that is generated directly to >> choose which environment is used for which PTransform. >> >> Can you provide additional details as to why you would want to have two >> separate java environments (e.g. incompatible versions of libraries)? >> >> On Wed, Sep 22, 2021 at 3:41 PM Ke Wu <ke.wu...@gmail.com> wrote: >> >> >> Thanks Luke for the reply, do you know what is the preferred way to >> configure a PTransform to be executed in a different environment from >> another PTransform when both are in the same SDK, e.g. Java ? >> >> Best, >> Ke >> >> On Sep 21, 2021, at 9:48 PM, Luke Cwik <lc...@google.com> wrote: >> >> Environments that aren't exactly the same are already in separate >> ExecutableStages. The GreedyPCollectionFuser ensures that today[1]. >> >> Workarounds like getOnlyEnvironmentId would need to be removed. It may also >> be effectively dead-code. >> >> 1: >> https://github.com/apache/beam/blob/ebf2aacf37b97fc85b167271f184f61f5b06ddc3/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java#L144 >> >> On Tue, Sep 21, 2021 at 1:45 PM Ke Wu <ke.wu...@gmail.com> wrote: >> >> >> Hello All, >> >> We have a use case where in a java portable pipeline, we would like to have >> multiple environments setup in order that some executable stage runs in one >> environment while some other executable stages runs in another environment. >> Couple of questions on this: >> >> 1. Is this current supported? I noticed a TODO in [1] which suggests it is >> feature pending support >> 2. If we did support it, what would the ideal mechanism to distinguish >> ParDo/ExecutableStage to be executed in different environment, is it through >> ResourceHints? >> >> >> Best, >> Ke >> >> >> [1] >> https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java#L344 >> >>
