2019년 4월 23일 (화) 오전 2:07, Robert Bradshaw <rober...@google.com>님이 작성:
> I've been out, so coming a bit late to the discussion, but here's my
> thoughts.
>
> The expansion service absolutely needs to be able to provide the
> dependencies for the transform(s) it expands. It seems the default,
> foolproof way of doing this is via the environment, which can be a
> docker image with all the required dependencies. More than this an
> (arguably important, but possibly messy) optimization.
>
> The standard way to provide artifacts outside of the environment is
> via the artifact staging service. Of course, the expansion service may
> not have access to the (final) artifact staging service (due to
> permissions, locality, or it may not even be started up yet) but the
> SDK invoking the expansion service could offer an artifact staging
> environment for the SDK to publish artifacts to. However, there are
> some difficulties here, in particular avoiding name collisions with
> staged artifacts, assigning semantic meaning to the artifacts (e.g.
> should jar files get automatically placed in the classpath, or Python
> packages recognized and installed at startup). The alternative is
> going with a (type, pointer) scheme for naming dependencies; if we go
> this route I think we should consider migrating all artifact staging
> to this style. I am concerned that the "file" version will be less
> than useful for what will become the most convenient expansion
> services (namely, hosted and docker image). I am still at a loss,
> however, as to how to solve the diamond dependency problem among
> dependencies--perhaps the information is there if one walks
> maven/pypi/go modules/... but do we expect every runner to know about
> every packaging platform? This also wouldn't solve the issue if fat
> jars are used as dependencies. The only safe thing to do here is to
> force distinct dependency sets to live in different environments,
> which could be too conservative.
>
> This all leads me to think that perhaps the environment itself should
> be docker image (often one of "vanilla" beam-java-x.y ones) +
> dependency list, rather than have the dependency/artifact list as some
> kind of data off to the side. In this case, the runner would (as
> requested by its configuration) be free to merge environments it
> deemed compatible, including swapping out beam-java-X for
> beam-java-embedded if it considers itself compatible with the
> dependency list.
Like this idea to build multiple docker environments on top of a bare
minimum SDK harness container and allow runners to pick a suitable one
based on a dependency list.
>
> I agree with Thomas that we'll want to make expansion services, and
> the transforms they offer, more discoverable. The whole lifetime cycle
> of expansion services is something that has yet to be fully fleshed
> out, and may influence some of these decisions.
>
> As for adding --jar_package to the Python SDK, this seems really
> specific to calling java-from-python (would we have O(n^2) such
> options?) as well as out-of-place for a Python user to specify. I
> would really hope we can figure out a more generic solution. If we
> need this option in the meantime, let's at least make it clear
> (probably in the name) that it's temporary.
>
Good points. I second that we need a more generic solution than
python-to-java specific option. I think instead of naming differently we
can make --jar_package a secondary option under --experiment in the
meantime. WDYT?
> On Tue, Apr 23, 2019 at 1:08 AM Thomas Weise <t...@apache.org> wrote:
> >
> > One more suggestion:
> >
> > It would be nice to be able to select the environment for the external
> transforms. For example, I would like to be able to use EMBEDDED for Flink.
> That's implicit for sources which are runner native unbounded read
> translations, but it should also be possible for writes. That would then be
> similar to how pipelines are packaged and run with the "legacy" runner.
> >
> > Thomas
> >
> >
> > On Mon, Apr 22, 2019 at 1:18 PM Ankur Goenka <goe...@google.com> wrote:
> >>
> >> Great discussion!
> >> I have a few points around the structure of proto but that is less
> important as it can evolve.
> >> However, I think that artifact compatibility is another important
> aspect to look at.
> >> Example: TransformA uses Guava 1.6>< 1.7, TransformB uses 1.8><1.9 and
> TransformC uses 1.6><1.8. As sdk provide the environment for each
> transform, it can not simply say EnvironmentJava for both TransformA and
> TransformB as the dependencies are not compatible.
> >> We should have separate environment associated with TransformA and
> TransformB in this case.
> >>
> >> To support this case, we need 2 things.
> >> 1: Granular metadata about the dependency including type.
> >> 2: Complete list of the transforms to be expanded.
> >>
> >> Elaboration:
> >> The compatibility check can be done in a crude way if we provide all
> the metadata about the dependency to expansion service.
> >> Also, the expansion service should expand all the applicable transforms
> in a single call so that it knows about incompatibility and create separate
> environments for these transforms. So in the above example, expansion
> service will associate EnvA to TransformA and EnvB to TransformB and EnvA
> to TransformC. This will ofcource require changes to Expansion service
> proto but giving all the information to expansion service will make it
> support more case and make it a bit more future proof.
> >>
> >>
> >> On Mon, Apr 22, 2019 at 10:16 AM Maximilian Michels <m...@apache.org>
> wrote:
> >>>
> >>> Thanks for the summary Cham. All makes sense. I agree that we want to
> >>> keep the option to manually specify artifacts.
> >>>
> >>> > There are few unanswered questions though.
> >>> > (1) In what form will a transform author specify dependencies ? For
> example, URL to a Maven repo, URL to a local file, blob ?
> >>>
> >>> Going forward, we probably want to support multiple ways. For now, we
> >>> could stick with a URL-based approach with support for different file
> >>> systems. In the future a list of packages to retrieve from Maven/PyPi
> >>> would be useful.
> >>>
> >> We can ask user for (type, metadata). For maven it can be something
> like (MAVEN, {groupId:com.google.guava, artifactId: guava, version: 19}) or
> (FILE, file://myfile)
> >> To begin with, we can only support a few types like File and can add
> more types in future.
> >>>
> >>> > (2) How will dependencies be included in the expansion response
> proto ? String (URL), bytes (blob) ?
> >>>
> >>> I'd go for a list of Protobuf strings first but the format would have
> to
> >>> evolve for other dependency types.
> >>>
> >> Here also (type, payload) should suffice. We can have interpreter for
> each type to translate the payload.
> >>>
> >>> > (3) How will we manage/share transitive dependencies required at
> runtime ?
> >>>
> >>> I'd say transitive dependencies have to be included in the list. In
> case
> >>> of fat jars, they are reduced to a single jar.
> >>
> >> Makes sense.
> >>>
> >>>
> >>> > (4) How will dependencies be staged for various runner/SDK
> combinations ? (for example, portable runner/Flink, Dataflow runner)
> >>>
> >>> Staging should be no different than it is now, i.e. go through Beam's
> >>> artifact staging service. As long as the protocol is stable, there
> could
> >>> also be different implementations.
> >>
> >> Makes sense.
> >>>
> >>>
> >>> -Max
> >>>
> >>> On 20.04.19 03:08, Chamikara Jayalath wrote:
> >>> > OK, sounds like this is a good path forward then.
> >>> >
> >>> > * When starting up the expansion service, user (that starts up the
> >>> > service) provide dependencies necessary to expand transforms. We will
> >>> > later add support for adding new transforms to an already running
> >>> > expansion service.
> >>> > * As a part of transform configuration, transform author have the
> option
> >>> > of providing a list of dependencies that will be needed to run the
> >>> > transform.
> >>> > * These dependencies will be send back to the pipeline SDK as a part
> of
> >>> > expansion response and pipeline SDK will stage these resources.
> >>> > * Pipeline author have the option of specifying the dependencies
> using a
> >>> > pipeline option. (for example,
> https://github.com/apache/beam/pull/8340)
> >>> >
> >>> > I think last option is important to (1) make existing transform
> easily
> >>> > available for cross-language usage without additional configurations
> (2)
> >>> > allow pipeline authors to override dependency versions specified by
> in
> >>> > the transform configuration (for example, to apply security patches)
> >>> > without updating the expansion service.
> >>> >
> >>> > There are few unanswered questions though.
> >>> > (1) In what form will a transform author specify dependencies ? For
> >>> > example, URL to a Maven repo, URL to a local file, blob ?
> >>> > (2) How will dependencies be included in the expansion response
> proto ?
> >>> > String (URL), bytes (blob) ?
> >>> > (3) How will we manage/share transitive dependencies required at
> runtime ?
> >>> > (4) How will dependencies be staged for various runner/SDK
> combinations
> >>> > ? (for example, portable runner/Flink, Dataflow runner)
> >>> >
> >>> > Thanks,
> >>> > Cham
> >>> >
> >>> > On Fri, Apr 19, 2019 at 4:49 AM Maximilian Michels <m...@apache.org
> >>> > <mailto:m...@apache.org>> wrote:
> >>> >
> >>> > Thank you for your replies.
> >>> >
> >>> > I did not suggest that the Expansion Service does the staging,
> but it
> >>> > would return the required resources (e.g. jars) for the external
> >>> > transform's runtime environment. The client then has to take
> care of
> >>> > staging the resources.
> >>> >
> >>> > The Expansion Service itself also needs resources to do the
> >>> > expansion. I
> >>> > assumed those to be provided when starting the expansion
> service. I
> >>> > consider it less important but we could also provide a way to
> add new
> >>> > transforms to the Expansion Service after startup.
> >>> >
> >>> > Good point on Docker vs externally provided environments. For
> the PR
> >>> > [1]
> >>> > it will suffice then to add Kafka to the container dependencies.
> The
> >>> > "--jar_package" pipeline option is ok for now but I'd like to
> see work
> >>> > towards staging resources for external transforms via information
> >>> > returned by the Expansion Service. That avoids users having to
> take
> >>> > care
> >>> > of including the correct jars in their pipeline options.
> >>> >
> >>> > These issues are related and we could discuss them in separate
> threads:
> >>> >
> >>> > * Auto-discovery of Expansion Service and its external transforms
> >>> > * Credentials required during expansion / runtime
> >>> >
> >>> > Thanks,
> >>> > Max
> >>> >
> >>> > [1] ttps://github.com/apache/beam/pull/8322
> >>> > <http://github.com/apache/beam/pull/8322>
> >>> >
> >>> > On 19.04.19 07:35, Thomas Weise wrote:
> >>> > > Good discussion :)
> >>> > >
> >>> > > Initially the expansion service was considered a user
> >>> > responsibility,
> >>> > > but I think that isn't necessarily the case. I can also see
> the
> >>> > > expansion service provided as part of the infrastructure and
> the
> >>> > user
> >>> > > not wanting to deal with it at all. For example, users may
> want
> >>> > to write
> >>> > > Python transforms and use external IOs, without being
> concerned how
> >>> > > these IOs are provided. Under such scenario it would be good
> if:
> >>> > >
> >>> > > * Expansion service(s) can be auto-discovered via the job
> service
> >>> > endpoint
> >>> > > * Available external transforms can be discovered via the
> expansion
> >>> > > service(s)
> >>> > > * Dependencies for external transforms are part of the
> metadata
> >>> > returned
> >>> > > by expansion service
> >>> > >
> >>> > > Dependencies could then be staged either by the SDK client or
> the
> >>> > > expansion service. The expansion service could provide the
> >>> > locations to
> >>> > > stage to the SDK, it would still be transparent to the user.
> >>> > >
> >>> > > I also agree with Luke regarding the environments. Docker is
> the
> >>> > choice
> >>> > > for generic deployment. Other environments are used when the
> >>> > flexibility
> >>> > > offered by Docker isn't needed (or gets into the way). Then
> the
> >>> > > dependencies are provided in different ways. Whether these are
> >>> > Python
> >>> > > packages or jar files, by opting out of Docker the decision is
> >>> > made to
> >>> > > manage dependencies externally.
> >>> > >
> >>> > > Thomas
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 6:01 PM Chamikara Jayalath
> >>> > <chamik...@google.com <mailto:chamik...@google.com>
> >>> > > <mailto:chamik...@google.com <mailto:chamik...@google.com>>>
> wrote:
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 5:21 PM Chamikara Jayalath
> >>> > > <chamik...@google.com <mailto:chamik...@google.com>
> >>> > <mailto:chamik...@google.com <mailto:chamik...@google.com>>>
> wrote:
> >>> > >
> >>> > > Thanks for raising the concern about credentials
> Ankur, I
> >>> > agree
> >>> > > that this is a significant issue.
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 4:23 PM Lukasz Cwik
> >>> > <lc...@google.com <mailto:lc...@google.com>
> >>> > > <mailto:lc...@google.com <mailto:lc...@google.com>>>
> wrote:
> >>> > >
> >>> > > I can understand the concern about credentials,
> the same
> >>> > > access concern will exist for several cross
> language
> >>> > > transforms (mostly IOs) since some will need
> access to
> >>> > > credentials to read/write to an external service.
> >>> > >
> >>> > > Are there any ideas on how credential propagation
> >>> > could work
> >>> > > to these IOs?
> >>> > >
> >>> > >
> >>> > > There are some cases where existing IO transforms need
> >>> > > credentials to access remote resources, for example,
> size
> >>> > > estimation, validation, etc. But usually these are
> >>> > optional (or
> >>> > > transform can be configured to not perform these
> functions).
> >>> > >
> >>> > >
> >>> > > To clarify, I'm only talking about transform expansion
> here.
> >>> > Many IO
> >>> > > transforms need read/write access to remote services at
> run
> >>> > time. So
> >>> > > probably we need to figure out a way to propagate these
> >>> > credentials
> >>> > > anyways.
> >>> > >
> >>> > > Can we use these mechanisms for staging?
> >>> > >
> >>> > >
> >>> > > I think we'll have to find a way to do one of (1)
> propagate
> >>> > > credentials to other SDKs (2) allow users to
> configure SDK
> >>> > > containers to have necessary credentials (3) do the
> artifact
> >>> > > staging from the pipeline SDK environment which
> already have
> >>> > > credentials. I prefer (1) or (2) since this will
> given a
> >>> > > transform same feature set whether used directly (in
> the same
> >>> > > SDK language as the transform) or remotely but it
> might
> >>> > be hard
> >>> > > to do this for an arbitrary service that a transform
> might
> >>> > > connect to considering the number of ways users can
> configure
> >>> > > credentials (after an offline discussion with Ankur).
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 3:47 PM Ankur Goenka
> >>> > > <goe...@google.com <mailto:goe...@google.com>
> >>> > <mailto:goe...@google.com <mailto:goe...@google.com>>> wrote:
> >>> > >
> >>> > > I agree that the Expansion service knows
> about the
> >>> > > artifacts required for a cross language
> transform and
> >>> > > having a prepackage folder/Zip for transforms
> >>> > based on
> >>> > > language makes sense.
> >>> > >
> >>> > > One think to note here is that expansion
> service
> >>> > might
> >>> > > not have the same access privilege as the
> pipeline
> >>> > > author and hence might not be able to stage
> >>> > artifacts by
> >>> > > itself.
> >>> > > Keeping this in mind I am leaning towards
> making
> >>> > > Expansion service provide all the required
> >>> > artifacts to
> >>> > > the user and let the user stage the artifacts
> as
> >>> > regular
> >>> > > artifacts.
> >>> > > At this time, we only have Beam File System
> based
> >>> > > artifact staging which users local
> credentials to
> >>> > access
> >>> > > different file systems. Even a docker based
> expansion
> >>> > > service running on local machine might not
> have
> >>> > the same
> >>> > > access privileges.
> >>> > >
> >>> > > In brief this is what I am leaning toward.
> >>> > > User call for pipeline submission -> Expansion
> >>> > service
> >>> > > provide cross language transforms and relevant
> >>> > artifacts
> >>> > > to the Sdk -> Sdk Submits the pipeline to
> >>> > Jobserver and
> >>> > > Stages user and cross language artifacts to
> artifacts
> >>> > > staging service
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 2:33 PM Chamikara
> Jayalath
> >>> > > <chamik...@google.com
> >>> > <mailto:chamik...@google.com> <mailto:chamik...@google.com
> >>> > <mailto:chamik...@google.com>>> wrote:
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 2:12 PM Lukasz
> Cwik
> >>> > > <lc...@google.com <mailto:
> lc...@google.com>
> >>> > <mailto:lc...@google.com <mailto:lc...@google.com>>> wrote:
> >>> > >
> >>> > > Note that Max did ask whether making
> the
> >>> > > expansion service do the staging made
> >>> > sense, and
> >>> > > my first line was agreeing with that
> >>> > direction
> >>> > > and expanding on how it could be done
> (so
> >>> > this
> >>> > > is really Max's idea or from whomever
> he
> >>> > got the
> >>> > > idea from).
> >>> > >
> >>> > >
> >>> > > +1 to what Max said then :)
> >>> > >
> >>> > >
> >>> > > I believe a lot of the value of the
> expansion
> >>> > > service is not having users need to be
> >>> > aware of
> >>> > > all the SDK specific dependencies when
> >>> > they are
> >>> > > trying to create a pipeline, only the
> >>> > "user" who
> >>> > > is launching the expansion service may
> >>> > need to.
> >>> > > And in that case we can have a
> prepackaged
> >>> > > expansion service application that
> does what
> >>> > > most users would want (e.g. expansion
> >>> > service as
> >>> > > a docker container, a single bundled
> jar,
> >>> > ...).
> >>> > > We (the Apache Beam community) could
> >>> > choose to
> >>> > > host a default implementation of the
> >>> > expansion
> >>> > > service as well.
> >>> > >
> >>> > >
> >>> > > I'm not against this. But I think this is
> a
> >>> > > secondary more advanced use-case. For a
> Beam
> >>> > users
> >>> > > that needs to use a Java transform that
> they
> >>> > already
> >>> > > have in a Python pipeline, we should
> provide
> >>> > a way
> >>> > > to allow starting up a expansion service
> (with
> >>> > > dependencies needed for that) and running
> a
> >>> > pipeline
> >>> > > that uses this external Java transform
> (with
> >>> > > dependencies that are needed at runtime).
> >>> > Probably,
> >>> > > it'll be enough to allow providing all
> >>> > dependencies
> >>> > > when starting up the expansion service
> and allow
> >>> > > expansion service to do the staging of
> jars are
> >>> > > well. I don't see a need to include the
> list
> >>> > of jars
> >>> > > in the ExpansionResponse sent to the
> Python SDK.
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 2:02 PM
> Chamikara
> >>> > > Jayalath <chamik...@google.com
> >>> > <mailto:chamik...@google.com>
> >>> > > <mailto:chamik...@google.com
> >>> > <mailto:chamik...@google.com>>> wrote:
> >>> > >
> >>> > > I think there are two kind of
> >>> > dependencies
> >>> > > we have to consider.
> >>> > >
> >>> > > (1) Dependencies that are needed
> to
> >>> > expand
> >>> > > the transform.
> >>> > >
> >>> > > These have to be provided when we
> >>> > start the
> >>> > > expansion service so that
> available
> >>> > external
> >>> > > transforms are correctly
> registered
> >>> > with the
> >>> > > expansion service.
> >>> > >
> >>> > > (2) Dependencies that are not
> needed at
> >>> > > expansion but may be needed at
> runtime.
> >>> > >
> >>> > > I think in both cases, users have
> to
> >>> > provide
> >>> > > these dependencies either when
> expansion
> >>> > > service is started or when a
> pipeline is
> >>> > > being executed.
> >>> > >
> >>> > > Max, I'm not sure why expansion
> >>> > service will
> >>> > > need to provide dependencies to
> the user
> >>> > > since user will already be aware
> of
> >>> > these.
> >>> > > Are you talking about a expansion
> service
> >>> > > that is readily available that
> will
> >>> > be used
> >>> > > by many Beam users ? I think such
> a
> >>> > > (possibly long running) service
> will
> >>> > have to
> >>> > > maintain a repository of
> transforms and
> >>> > > should have mechanism for
> registering new
> >>> > > transforms and discovering already
> >>> > > registered transforms etc. I think
> >>> > there's
> >>> > > more design work needed to make
> transform
> >>> > > expansion service support such
> use-cases.
> >>> > > Currently, I think allowing
> pipeline
> >>> > author
> >>> > > to provide the jars when starting
> the
> >>> > > expansion service and when
> executing the
> >>> > > pipeline will be adequate.
> >>> > >
> >>> > > Regarding the entity that will
> >>> > perform the
> >>> > > staging, I like Luke's idea of
> allowing
> >>> > > expansion service to do the
> staging
> >>> > (of jars
> >>> > > provided by the user). Notion of
> >>> > artifacts
> >>> > > and how they are
> extracted/represented is
> >>> > > SDK dependent. So if the pipeline
> SDK
> >>> > tries
> >>> > > to do this we have to add n x (n
> -1)
> >>> > > configurations (for n SDKs).
> >>> > >
> >>> > > - Cham
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 11:45 AM
> >>> > Lukasz Cwik
> >>> > > <lc...@google.com
> >>> > <mailto:lc...@google.com> <mailto:lc...@google.com
> >>> > <mailto:lc...@google.com>>>
> >>> > > wrote:
> >>> > >
> >>> > > We can expose the artifact
> staging
> >>> > > endpoint and artifact token to
> >>> > allow the
> >>> > > expansion service to upload
> any
> >>> > > resources its environment may
> >>> > need. For
> >>> > > example, the expansion service
> >>> > for the
> >>> > > Beam Java SDK would be able to
> >>> > upload jars.
> >>> > >
> >>> > > In the "docker" environment,
> the
> >>> > Apache
> >>> > > Beam Java SDK harness
> container would
> >>> > > fetch the relevant artifacts
> for
> >>> > itself
> >>> > > and be able to execute the
> pipeline.
> >>> > > (Note that a docker
> environment could
> >>> > > skip all this artifact
> staging if the
> >>> > > docker environment contained
> all
> >>> > > necessary artifacts).
> >>> > >
> >>> > > For the existing "external"
> >>> > environment,
> >>> > > it should already come with
> all the
> >>> > > resources prepackaged wherever
> >>> > > "external" points to. The
> "process"
> >>> > > based environment could
> choose to use
> >>> > > the artifact staging service
> to fetch
> >>> > > those resources associated
> with its
> >>> > > process or it could follow
> the same
> >>> > > pattern that "external" would
> do and
> >>> > > already contain all the
> prepackaged
> >>> > > resources. Note that both
> >>> > "external" and
> >>> > > "process" will require the
> >>> > instance of
> >>> > > the expansion service to be
> >>> > specialized
> >>> > > for those environments which
> is
> >>> > why the
> >>> > > default should for the
> expansion
> >>> > service
> >>> > > to be the "docker"
> environment.
> >>> > >
> >>> > > Note that a major reason for
> >>> > going with
> >>> > > docker containers as the
> environment
> >>> > > that all runners should
> support
> >>> > is that
> >>> > > containers provides a solution
> >>> > for this
> >>> > > exact issue. Both the
> "process" and
> >>> > > "external" environments are
> >>> > explicitly
> >>> > > limiting and expanding their
> >>> > > capabilities will quickly
> have us
> >>> > > building something like a
> docker
> >>> > > container because we'll
> quickly find
> >>> > > ourselves solving the same
> >>> > problems that
> >>> > > docker containers provide
> (resources,
> >>> > > file layout, permissions, ...)
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, Apr 18, 2019 at 11:21
> AM
> >>> > > Maximilian Michels
> >>> > <m...@apache.org <mailto:m...@apache.org>
> >>> > > <mailto:m...@apache.org
> >>> > <mailto:m...@apache.org>>> wrote:
> >>> > >
> >>> > > Hi everyone,
> >>> > >
> >>> > > We have previously merged
> support
> >>> > > for configuring
> transforms across
> >>> > > languages. Please see
> Cham's
> >>> > summary
> >>> > > on the discussion [1].
> There is
> >>> > > also a design document
> [2].
> >>> > >
> >>> > > Subsequently, we've added
> >>> > wrappers
> >>> > > for cross-language
> transforms
> >>> > to the
> >>> > > Python SDK, i.e.
> >>> > GenerateSequence,
> >>> > > ReadFromKafka, and there
> is a
> >>> > pending
> >>> > > PR [1] for WriteToKafka.
> All
> >>> > of them
> >>> > > utilize Java transforms
> via
> >>> > > cross-language
> configuration.
> >>> > >
> >>> > > That is all pretty
> exciting :)
> >>> > >
> >>> > > We still have some issues
> to
> >>> > solve,
> >>> > > one being how to stage
> >>> > artifact from
> >>> > > a foreign environment.
> When
> >>> > we run
> >>> > > external transforms which
> are
> >>> > part of
> >>> > > Beam's core (e.g.
> >>> > GenerateSequence),
> >>> > > we have them available in
> the SDK
> >>> > > Harness. However, when
> they
> >>> > are not
> >>> > > (e.g. KafkaIO) we need to
> >>> > stage the
> >>> > > necessary files.
> >>> > >
> >>> > > For my PR [3] I've
> naively added
> >>> > >
> ":beam-sdks-java-io-kafka" to
> >>> > the SDK
> >>> > > Harness which caused
> dependency
> >>> > > problems [4]. Those could
> be
> >>> > resolved
> >>> > > but the bigger question
> is how to
> >>> > > stage artifacts for
> external
> >>> > > transforms
> programmatically?
> >>> > >
> >>> > > Heejong has solved this by
> >>> > adding a
> >>> > > "--jar_package" option to
> the
> >>> > Python
> >>> > > SDK to stage Java files
> [5].
> >>> > I think
> >>> > > that is a better solution
> than
> >>> > > adding required Jars to
> the SDK
> >>> > > Harness directly, but it
> is
> >>> > not very
> >>> > > convenient for users.
> >>> > >
> >>> > > I've discussed this today
> with
> >>> > > Thomas and we both figured
> >>> > that the
> >>> > > expansion service needs to
> >>> > provide a
> >>> > > list of required Jars
> with the
> >>> > > ExpansionResponse it
> >>> > provides. It's
> >>> > > not entirely clear, how we
> >>> > determine
> >>> > > which artifacts are
> necessary
> >>> > for an
> >>> > > external transform. We
> could just
> >>> > > dump the entire classpath
> >>> > like we do
> >>> > > in PipelineResources for
> Java
> >>> > > pipelines. This provides
> many
> >>> > > unneeded classes but
> would work.
> >>> > >
> >>> > > Do you think it makes
> sense
> >>> > for the
> >>> > > expansion service to
> provide the
> >>> > > artifacts? Perhaps you
> have a
> >>> > better
> >>> > > idea how to resolve the
> staging
> >>> > > problem in cross-language
> >>> > pipelines?
> >>> > >
> >>> > > Thanks,
> >>> > > Max
> >>> > >
> >>> > > [1]
> >>> > >
> >>> >
> https://lists.apache.org/thread.html/b99ba8527422e31ec7bb7ad9dc3a6583551ea392ebdc5527b5fb4a67@%3Cdev.beam.apache.org%3E
> >>> > >
> >>> > > [2]
> >>> > > https://s.apache.org/beam-cross-language-io
> >>> > >
> >>> > > [3]
> >>> > >
> https://github.com/apache/beam/pull/8322#discussion_r276336748
> >>> > >
> >>> > > [4] Dependency graph for
> >>> > > beam-runners-direct-java:
> >>> > >
> >>> > > beam-runners-direct-java
> ->
> >>> > > sdks-java-harness ->
> >>> > > beam-sdks-java-io-kafka
> >>> > > ->
> beam-runners-direct-java
> >>> > ... the
> >>> > > cycle continues
> >>> > >
> >>> > > Beam-runners-direct-java
> >>> > depends on
> >>> > > sdks-java-harness due
> >>> > > to the infamous Universal
> Local
> >>> > > Runner.
> >>> > Beam-sdks-java-io-kafka depends
> >>> > > on
> beam-runners-direct-java for
> >>> > > running tests.
> >>> > >
> >>> > > [5]
> >>> > > https://github.com/apache/beam/pull/8340
> >>> > >
> >>> >
>
>
