Re: (mini-doc) Beam (Flink) portable job templates

Thanks for the follow up, Thomas. On Mon, Oct 28, 2019 at 7:55 PM Thomas Weise wrote: > Follow-up for users looking to run portable pipelines on Flink: > > After prototyping the generate-jar-file approach for internal deployment > and some related discussion, the conclusion was that it is too

Re: (mini-doc) Beam (Flink) portable job templates

Follow-up for users looking to run portable pipelines on Flink: After prototyping the generate-jar-file approach for internal deployment and some related discussion, the conclusion was that it is too limiting. The sticky point is that the jar file would need to be generated at container build

Re: (mini-doc) Beam (Flink) portable job templates

Following up on discussion in this morning's OSS runners meeting, I have uploaded a draft PR for the full implementation (job creation + execution): https://github.com/apache/beam/pull/9408 Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Tue, Aug 20, 2019 at 1:24 PM

Re: (mini-doc) Beam (Flink) portable job templates

On Tue, Aug 20, 2019 at 8:56 AM Lukasz Cwik wrote: > > > On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay wrote: > >> >> >> On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote: >> >>> There is a PR open for this: https://github.com/apache/beam/pull/9331 >>> >>> (it wasn't tagged with the JIRA and

Re: (mini-doc) Beam (Flink) portable job templates

On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay wrote: > > > On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote: > >> There is a PR open for this: https://github.com/apache/beam/pull/9331 >> >> (it wasn't tagged with the JIRA and therefore not linked) >> >> I think it is worthwhile to explore how

Re: (mini-doc) Beam (Flink) portable job templates

On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay wrote: > > Can we come up with a partially constructed proto that can be produced by >> just running the Python entry point? Note this would also require pushing >> the pipeline options parsing into the job service. >> > > Why would this require

Re: (mini-doc) Beam (Flink) portable job templates

On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise wrote: > There is a PR open for this: https://github.com/apache/beam/pull/9331 > > (it wasn't tagged with the JIRA and therefore not linked) > > I think it is worthwhile to explore how we could further detangle the > client side Python and Java

Re: (mini-doc) Beam (Flink) portable job templates

There is a PR open for this: https://github.com/apache/beam/pull/9331 (it wasn't tagged with the JIRA and therefore not linked) I think it is worthwhile to explore how we could further detangle the client side Python and Java dependencies. The expansion service is one more dependency to

Re: (mini-doc) Beam (Flink) portable job templates

I found the tracking ticket at BEAM-7966 On Sun, Aug 18, 2019 at 11:59 AM enrico canzonieri wrote: > Is this alternative still being considered? Creating a portable jar sounds > like a good solution to re-use the existing runner specific

Re: (mini-doc) Beam (Flink) portable job templates

Is this alternative still being considered? Creating a portable jar sounds like a good solution to re-use the existing runner specific deployment mechanism (e.g. Flink k8s operator) and in general simplify the deployment story. On Fri, Aug 9, 2019 at 12:46 AM Robert Bradshaw wrote: > The

Re: (mini-doc) Beam (Flink) portable job templates

The expansion service is a separate service. (The flink jar happens to bring both up.) However, there is negotiation to receive/validate the pipeline options. On Fri, Aug 9, 2019 at 1:54 AM Thomas Weise wrote: > > We would also need to consider cross-language pipelines that (currently) > assume

Re: (mini-doc) Beam (Flink) portable job templates

We would also need to consider cross-language pipelines that (currently) assume the interaction with an expansion service at construction time. On Thu, Aug 8, 2019, 4:38 PM Kyle Weaver wrote: > > It might also be useful to have the option to just output the proto and > artifacts, as alternative

Re: (mini-doc) Beam (Flink) portable job templates

> It might also be useful to have the option to just output the proto and artifacts, as alternative to the jar file. Sure, that wouldn't be too big a change if we were to decide to go the SDK route. > For the Flink entry point we would need to allow for the job server to be used as a library.

Re: (mini-doc) Beam (Flink) portable job templates

I also added this as option for pipeline submission to the k8s discussion: https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.iov21d695qx5 On Thu, Aug 8, 2019 at 4:21 PM Thomas Weise wrote: > Hi Kyle, > > It might also be useful to have the option

Re: (mini-doc) Beam (Flink) portable job templates

Hi Kyle, It might also be useful to have the option to just output the proto and artifacts, as alternative to the jar file. For the Flink entry point we would need to allow for the job server to be used as a library. It would probably not be too hard to have the Flink job constructed via the

Re: (mini-doc) Beam (Flink) portable job templates

Re Javaless/serverless solution: I take it this would probably mean that we would construct the jar directly from the SDK. There are advantages to this: full separation of Python and Java environments, no need for a job server, and likely a simpler implementation, since we'd no longer have to work

Re: (mini-doc) Beam (Flink) portable job templates

On Thu, Aug 8, 2019 at 8:29 AM Robert Bradshaw wrote: > > Before assembling the jar, the job server runs to create the > ingredients. That requires the (matching) Java environment on the Python > developers machine. > > We can run the job server and have it create the jar (and if we keep > the

Re: (mini-doc) Beam (Flink) portable job templates

On Wed, Aug 7, 2019 at 5:59 PM Thomas Weise wrote: > >> > * The pipeline construction code itself may need access to cluster >> > resources. In such cases the jar file cannot be created offline. >> >> Could you elaborate? > > > The entry point is arbitrary code written by the user, not limited

Re: (mini-doc) Beam (Flink) portable job templates

> For example, there could be access to a file system or other service to fetch metadata that is required to build the pipeline. That's a good point. It's totally up to users to decide how they want to deploy. I just think the jar solution would provide a useful option for many, but not all use

Re: (mini-doc) Beam (Flink) portable job templates

--> > > > > * The pipeline construction code itself may need access to cluster > resources. In such cases the jar file cannot be created offline. > > Could you elaborate? > The entry point is arbitrary code written by the user, not limited to Beam pipeline construction alone. For example, there

Re: (mini-doc) Beam (Flink) portable job templates

On Wed, Aug 7, 2019 at 6:20 AM Thomas Weise wrote: > > Hi Kyle, > > [document doesn't have comments enabled currently] > > As noted, worker deployment is an open question. I believe pipeline > submission and worker execution need to be considered together for a complete > deployment story. The

Re: (mini-doc) Beam (Flink) portable job templates

Hi Kyle, [document doesn't have comments enabled currently] As noted, worker deployment is an open question. I believe pipeline submission and worker execution need to be considered together for a complete deployment story. The idea of creating a self containing jar file is interesting, but

(mini-doc) Beam (Flink) portable job templates

Hi all, Following up on discussion about portable Beam on Flink on Kubernetes [1], I have drafted a short document on how I propose we bundle portable Beam applications into jars that can be run on OSS runners, similar to Dataflow templates (but without the actual template part, at least for the