On Wed, Aug 7, 2019 at 6:20 AM Thomas Weise <[email protected]> wrote: > > Hi Kyle, > > [document doesn't have comments enabled currently] > > As noted, worker deployment is an open question. I believe pipeline > submission and worker execution need to be considered together for a complete > deployment story. The idea of creating a self containing jar file is > interesting, but there are trade-offs: > > * The pipeline construction code itself may need access to cluster resources. > In such cases the jar file cannot be created offline.
Could you elaborate? > * For k8s deployment, a container image with the SDK and application code is > required for the worker. The jar file (which is really a derived artifact) > would need to be built in addition to the container image. Yes. For standard use, a vanilla released Beam published SDK container + staged artifacts should be sufficient. > * To build such jar file, the user would need a build environment with job > server and application code. Do we want to make that assumption? Actually, it's probably much easier than that. A jar file is just a zip file with a standard structure, to which one can easily add (data) files without having a full build environment. The (pre-compiled) main class would know how to read this data to construct the pipeline and kick off the job just like any other Flink job. > The document that I had shared discusses options for pipeline submission. It > might be interesting to explore if your proposal for building such a jar can > be integrated or if you have other comments? > > Thomas > > > > On Tue, Aug 6, 2019 at 5:03 PM Kyle Weaver <[email protected]> wrote: >> >> Hi all, >> >> Following up on discussion about portable Beam on Flink on Kubernetes [1], I >> have drafted a short document on how I propose we bundle portable Beam >> applications into jars that can be run on OSS runners, similar to Dataflow >> templates (but without the actual template part, at least for the first >> iteration). It's pretty straightforward, but I thought I would broadcast it >> here in case anyone is interested. >> >> https://docs.google.com/document/d/1kj_9JWxGWOmSGeZ5hbLVDXSTv-zBrx4kQRqOq85RYD4/edit# >> >> [1] >> https://lists.apache.org/thread.html/a12dd939c4af254694481796bc08b05bb1321cfaadd1a79cd3866584@%3Cdev.beam.apache.org%3E >> >> Kyle Weaver | Software Engineer | github.com/ibzib | [email protected]
