Re Javaless/serverless solution: I take it this would probably mean that we would construct the jar directly from the SDK. There are advantages to this: full separation of Python and Java environments, no need for a job server, and likely a simpler implementation, since we'd no longer have to work within the constraints of the existing job server infrastructure. The only downside I can think of is the additional cost of implementing/maintaining jar creation code in each SDK, but that cost may be acceptable if it's simple enough.
Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Thu, Aug 8, 2019 at 9:31 AM Thomas Weise <t...@apache.org> wrote: > > > On Thu, Aug 8, 2019 at 8:29 AM Robert Bradshaw <rober...@google.com> > wrote: > >> > Before assembling the jar, the job server runs to create the >> ingredients. That requires the (matching) Java environment on the Python >> developers machine. >> >> We can run the job server and have it create the jar (and if we keep >> the job server running we can use it to interact with the running >> job). However, if the jar layout is simple enough, there's no need to >> even build it from Java. >> >> Taken to the extreme, this is a one-shot, jar-based JobService API. We >> choose a standard layout of where to put the pipeline description and >> artifacts, and can "augment" an existing jar (that has a >> runner-specific main class whose entry point knows how to read this >> data to kick off a pipeline as if it were a users driver code) into >> one that has a portable pipeline packaged into it for submission to a >> cluster. >> > > It would be nice if the Python developer doesn't have to run anything Java > at all. > > As we just discussed offline, this could be accomplished by including the > proto that is produced by the SDK into the pre-existing jar. > > And if the jar has an entry point that creates the Flink job in the > prescribed manner [1], it can be directly submitted to the Flink REST API. > That would allow for Java free client. > > [1] > https://lists.apache.org/thread.html/6db869c53816f4e2917949a7c6992c2b90856d7d639d7f2e1cd13768@%3Cdev.flink.apache.org%3E > >
