On Thu, Aug 8, 2019 at 8:29 AM Robert Bradshaw <[email protected]> wrote:
> > Before assembling the jar, the job server runs to create the > ingredients. That requires the (matching) Java environment on the Python > developers machine. > > We can run the job server and have it create the jar (and if we keep > the job server running we can use it to interact with the running > job). However, if the jar layout is simple enough, there's no need to > even build it from Java. > > Taken to the extreme, this is a one-shot, jar-based JobService API. We > choose a standard layout of where to put the pipeline description and > artifacts, and can "augment" an existing jar (that has a > runner-specific main class whose entry point knows how to read this > data to kick off a pipeline as if it were a users driver code) into > one that has a portable pipeline packaged into it for submission to a > cluster. > It would be nice if the Python developer doesn't have to run anything Java at all. As we just discussed offline, this could be accomplished by including the proto that is produced by the SDK into the pre-existing jar. And if the jar has an entry point that creates the Flink job in the prescribed manner [1], it can be directly submitted to the Flink REST API. That would allow for Java free client. [1] https://lists.apache.org/thread.html/6db869c53816f4e2917949a7c6992c2b90856d7d639d7f2e1cd13768@%3Cdev.flink.apache.org%3E
