Le mar. 8 mai 2018 10:16, Robert Bradshaw <rober...@google.com> a écrit :
> On Sun, May 6, 2018 at 1:30 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > Wow, this mail should be on the website Robert, thanks for it > > > I still have a point to try to understand better: my view is that once > submitted the only perf related point is when you hit a flow of data. So a > split can be slow bit it is not a that big deal. So a runner integration > only needs to optimize process and nextElement logics, right? > > Yes. In some streaming cases (e.g. microbatch like Spark or Dataflow) there > may be many, many bundles, so the "control plane" part can't be /too/ slow, > but it's not as performance critical. > > > It is almost always doable to batch that - with triggers and other > constraints. So the portable model is elegant but not done to be "fast" in > current state of impl. > > Actually batching and streaming RPCs for the data plane has been there from > the start, for these reasons. > > > So this all leads to 2 needs: > > > 1. Have some native runner for dev > > 2. Have some bulk api for prod > > > In all cases this is decoralated of any runner no? Can even be a beam > subproject built on top of beam which would be very sane and ensure a clear > separation of concerns no? > > The thing to do here would be to extend the Environment (message) to allow > for alternatives, and then abstract out the creation of an bundle executor > such that different once could be instantiated based on this environment. > Agree so we need a generic runner delegating to "subrunners" (or runner impl) instead of impl-ing it in all runners. Sounds very sane, scalable and extensible/composable this way. Can we mark it as a backlog item and goal? > > Le 6 mai 2018 00:59, "Robert Bradshaw" <rober...@google.com> a écrit : > > >> Portability, at its core, is providing a spec for any runner to talk to > any > >> SDK. I personally think it's done a great job in cleaning up the model > by > >> forcing us to define a clean boundary (as specified at > >> https://github.com/apache/beam/tree/master/model ) between these two > >> components (even if the implementations of one or the other are > >> (temporarily, I hope for the most part) complicated).The pipeline (on > the > >> runner submission side) and work execution (on what has traditionally > been > >> called the fn api side) have concrete platform-independent descriptions, > >> rather than being a set of Java classes. > > >> Currently, the portion that lives on the "runner" side of this boundary > is > >> often shared among Java runners (via libraries like runners core), but > it > >> is all still part of each runner, and because of this it removes the > >> requirement for the Runner be Java just like it remove the requirement > for > >> the SDK to speak Java. (For example, I think a Python Dask runner makes > a > >> lot of sense, Dataflow may decide to implement larger portions of its > >> runner in Go or C++ or even behind a service, and I've used the Python > >> ULRunner to run the Java SDK over the Fn API for testing development > >> purposes). > > >> There is also the question of "why docker?" I actually don't see docker > all > >> that intrinsic to the protocol; one only needs to be able to define and > >> bring up workers that communicate on specified ports. Docker happens to > be > >> a fairly well supported way to package up an arbitrary chunk of code (in > >> any language), together with its nearly arbitrarily specified > >> dependencies/environment, in a way that's well specified and easy to > start > >> up. > > >> I would welcome changes to > > > https://github.com/apache/beam/blob/v2.4.0/model/pipeline/src/main/proto/beam_runner_api.proto#L730 > >> that would provide alternatives to docker (one of which comes to mind is > "I > >> already brought up a worker(s) for you (which could be the same process > >> that handled pipeline construction in testing scenarios), here's how to > >> connect to it/them.") Another option, which would seem to appeal to you > in > >> particular, would be "the worker code is linked into the runner's > binary, > >> use this process as the worker" (though note even for java-on-java, it > can > >> be advantageous to shield the worker and runner code from each others > >> environments, dependencies, and version requirements.) This latter > should > >> still likely use the FnApi to talk to itself (either over GRPC on local > >> ports, or possibly better via direct function calls eliminating the RPC > >> overhead altogether--this is how the fast local runner in Python works). > >> There may be runner environments well controlled enough that "start up > the > >> workers" could be specified as "run this command line." We should make > this > >> environment message extensible to other alternatives than "docker > container > >> url," though of course we don't want the set of options to grow too > large > >> or we loose the promise of portability unless every runner supports > every > >> protocol. > > >> Of course, the runner is always free to execute any Fn for which it > >> completely understands the URN and the environment any way it pleases, > e.g. > >> directly in process, or even via lighter-weight mechanism like Jython or > >> Graal, rather than asking an external process to do it. But we need a > >> lowest common denominator for executing arbitrary URNs runners are not > >> expected to understand. > > >> As an aside, there are also technical limitations in implementing > >> Portability > >> by simply requiring all runners to be Java and the portable layer simply > >> being wrappers of UserFnInLangaugeX in an equivalent UserFnObjectInJava, > >> executing everything as if it were pure Java. In particular the > overheads > >> of unnecessarily crossing the language boundaries many times in a single > >> fused graph are often prohibitive. > > >> Sorry for the long email, but hopefully this helps shed some light on > (at > >> least how I see) the portability effort (at the core of the Beam vision > >> statement) as well as concrete actions we can take to decouple it from > >> specific technologies. > > >> - Robert > > > >> On Sat, May 5, 2018 at 2:06 PM Romain Manni-Bucau < > rmannibu...@gmail.com> > >> wrote: > > >> > All are good points. > > >> > The only "?" I keep is: why beam doesnt uses its visitor api to make > the > >> portability transversal to all runners "mutating" the user model before > >> translation? Technically it sounds easy and avoid hacking all impl. Was > it > >> tested and failed? > > >> > Le 5 mai 2018 22:50, "Thomas Weise" <t...@apache.org> a écrit : > > >> >> Docker isn't a silver bullet and may not be the best choice for all > >> environments (I'm also looking at potentially launching SDK workers in a > >> different way), but AFAIK there has not been any alternative proposal > for > >> default SDK execution that can handle all of Python, Go and Java. > > >> >> Regardless of the default implementation, we should strive to keep > the > >> implementation modular so users can plug in their own replacement as > >> needed. Looking at the prototype implementation, Docker comes downstream > of > >> FlinkExecutableStageFunction, and it will be possible to supply a custom > >> implementation by making the translator pluggable (which I intend to > work > >> on once backporting to master is complete), and possibly > >> "SDKHarnessManager" itself can also be swapped out. > > >> >> I would also prefer that for Flink and other Java based runners we > >> retain the option to inline executable stages that are in Java. I would > >> expect a good number of use cases to benefit from direct execution in > the > >> task manager, and it may be good to offer the user that optimization. > > >> >> Thanks, > >> >> Thomas > > > > >> >> On Sat, May 5, 2018 at 12:54 PM, Eugene Kirpichov < > kirpic...@google.com> > >> wrote: > > >> >>> To add on that: Romain, if you are really excited about Graal as a > >> project, here are some constructive suggestions as to what you can do on > a > >> reasonably short timeframe: > >> >>> - Propose/prototype a design for writing UDFs in Beam SQL using > Graal > >> >>> - Go through the portability-related design documents, come up with > a > >> more precise assessment of what parts are actually dependent on Docker's > >> container format and/or on Docker itself, and propose a plan for > untangling > >> this dependency and opening the door to other mechanisms of > cross-language > >> execution > > >> >>> On Sat, May 5, 2018 at 12:50 PM Eugene Kirpichov < > kirpic...@google.com> > >> wrote: > > >> >>>> Graal is a very young project, currently nowhere near the level of > >> maturity or completeness as to be sufficient for Beam to fully bet its > >> portability vision on it: > >> >>>> - Graal currently only claims to support Java and Javascript, with > >> Ruby and R in the status of "some applications may run", Python support > >> "just beginning", and Go lacking altogether. > >> >>>> - Regarding existing production usage, the Graal FAQ says it is "a > >> project with new innovative technology in its early stages." > > >> >>>> That said, as Graal matures, I think it would be reasonable to keep > an > >> eye on it as a potential future lightweight alternative to containers > for > >> pipelines where Graal's level of support is sufficient for this > particular > >> pipeline. > > >> >>>> Please also keep in mind that execution of user code is only a > small > >> part of the overall portability picture, and dependency on Docker is an > >> even smaller part of that (there is only 1 mention of the word "Docker" > in > >> all of Beam's portability protos, and the mention is in an out-of-date > TODO > >> comment). I hope this addresses your concerns. > > >> >>>> On Sat, May 5, 2018 at 11:49 AM Romain Manni-Bucau < > >> rmannibu...@gmail.com> wrote: > > >> >>>>> Agree > > >> >>>>> The jvm is still mainstream for big data and it is trivial to have > a > >> remote facade to support natives but no point to have it in runners, it > is > >> some particular transforms or even dofn and sources only... > > > >> >>>>> Le 5 mai 2018 19:03, "Andrew Pilloud" <apill...@google.com> a > écrit : > > >> >>>>>> Thanks for the examples earlier, I think Hazelcast is a great > >> example of something portability might make more difficult. I'm not > working > >> on portability, but my understanding is that the data sent to the runner > is > >> a blob of code and the name of the container to run it in. A runner with > a > >> native language (java on Hazelcast for example) could run the code > directly > >> without the container if it is in a language it supports. So when > Hazelcast > >> sees a known java container specified, it just loads the java blob and > runs > >> it. When it sees another container it rejects the pipeline. You could > use > >> Graal in the Hazelcast runner to do this for a number of languages. I > would > >> expect that this could also be done in the direct runner, which > similarly > >> provides a native java environment, so portable Java pipelines can be > >> tested without docker? > > >> >>>>>> For another way to frame this: if Beam was originally written in > Go, > >> we would be having a different discussion. A pipeline written entirely > in > >> java wouldn't be possible, so instead to enable Hazelcast, we would have > to > >> be able to run the java from portability without running the container. > > >> >>>>>> Andrew > > >> >>>>>> On Sat, May 5, 2018 at 1:48 AM Romain Manni-Bucau < > >> rmannibu...@gmail.com> wrote: > > > > >> >>>>>>> 2018-05-05 9:27 GMT+02:00 Ismaël Mejía <ieme...@gmail.com>: > > >> >>>>>>>> Graal would not be a viable solution for the reasons Henning > and > >> Andrew > >> >>>>>>>> mentioned, or put in other words, when users choose a > programming > >> language > >> >>>>>>>> they don’t choose only a ‘friendly’ syntax or programming > model, > >> they > >> >>>>>>>> choose also the ecosystem that comes with it, and the libraries > >> that make > >> >>>>>>>> their life easier. However isolating these user > >> libraries/dependencies is a > >> >>>>>>>> hard problem and so far the standard solution to this problem > is > >> to use > >> >>>>>>>> operating systems containers via docker. > > > >> >>>>>>> Graal solves that Ismael. Same kind of experience than running > npm > >> libs on nashorn but with a more unified API to run any language soft. > > > > >> >>>>>>>> The Beam vision from day zero is to run pipelines written in > >> multiple > >> >>>>>>>> languages in runners in multiple systems, and so far we are not > >> doing this > >> >>>>>>>> in particular in the Apache runners. The portability work is > the > >> cleanest > >> >>>>>>>> way to achieve this vision given the constraints. > > > >> >>>>>>> Hmm, did I read it wrong and we don't have specific integration > of > >> the portable API in runners? This is what is messing up the runners and > >> limiting beam adoption on existing runners. > >> >>>>>>> Portable API is a feature buildable on top of runner, not in > >> runners. > >> >>>>>>> Same as a runner implementing the 5-6 primitives can run > anything, > >> the portable API should just rely on that and not require more > integration. > >> >>>>>>> It doesn't prevent more deep integrations as for some higher > level > >> primitives existing in runners but it is not the case today for runners > so > >> shouldn't exist IMHO. > > > > >> >>>>>>>> I agree however that for the Java SDK to Java runner case this > can > >> >>>>>>>> represent additional pain, docker ideally should not be a > >> requirement for > >> >>>>>>>> Java users with the Direct runner and debugging a pipeline > should > >> be as > >> >>>>>>>> easy as it is today. I think the Univerrsal Local Runner exists > to > >> cover > >> >>>>>>>> the Portable case, but after looking at this JIRA I am not sure > if > >> >>>>>>>> unification is coming (and by consequence if docker would be > >> mandatory). > >> >>>>>>>> https://issues.apache.org/jira/browse/BEAM-4239 > > >> >>>>>>>> I suppose for the distributed runners that they must implement > the > >> full > >> >>>>>>>> Portability APIs to be considered Beam multi language compliant > >> but they > >> >>>>>>>> can prefer for performance reasons to translate without the > >> portability > >> >>>>>>>> APIs the Java to Java case. > > > > >> >>>>>>> This is my issue, language portability must NOT impact runners > at > >> all, it is just a way to forward primitives to a runner. > >> >>>>>>> See it as a layer rewriting the pipeline and submitting it. No > need > >> to modify any runner. > > > >> >>>>>>>> On Sat, May 5, 2018 at 9:11 AM Reuven Lax <re...@google.com> > wrote: > > >> >>>>>>>> > A beam cluster with the spark runner would include a spark > >> cluster, plus > >> >>>>>>>> what's needed for portability, plus the beam sdk. > > >> >>>>>>>> > On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau < > >> rmannibu...@gmail.com> > >> >>>>>>>> wrote: > > > > >> >>>>>>>> >> Le 5 mai 2018 08:43, "Reuven Lax" <re...@google.com> a > écrit > : > > >> >>>>>>>> >> I don't believe we enforce docker anywhere. In fact if > someone > >> wanted to > >> >>>>>>>> run an all-windows beam cluster, they would probably not use > >> docker for > >> >>>>>>>> their runner (docker runs on Windows, but not efficiently). > > > > >> >>>>>>>> >> Or doesnt run sometimes - a colleague hit that yesterday :(. > > >> >>>>>>>> >> What is a "beam cluster" - opposed to a spark or foink > cluster? > >> How > >> >>>>>>>> would it work on windows servers? > > > >> >>>>>>>> >> On Fri, May 4, 2018, 11:19 PM Romain Manni-Bucau < > >> rmannibu...@gmail.com> > >> >>>>>>>> wrote: > > > > >> >>>>>>>> >>> 2018-05-05 2:33 GMT+02:00 Andrew Pilloud < > apill...@google.com>: > > >> >>>>>>>> >>>> What docker really buys is a package format and runtime > >> environment > >> >>>>>>>> that is language and operating system agnostic. The docker > >> packaging and > >> >>>>>>>> runtime format is the de facto standard for portable > applications > >> such as > >> >>>>>>>> this, and there is a group trying to turn it into an actual > >> standard. > > >> >>>>>>>> >>>> I would agree with you that dockerd has become bloated but > >> there are > >> >>>>>>>> projects that solve that. There is no longer lock-in to > dockerd, > >> there are > >> >>>>>>>> package format compatible docker replacements that eliminate > the > >> >>>>>>>> performance issues and overhead associated with docker. CRI-O ( > >> >>>>>>>> https://github.com/kubernetes-incubator/cri-o) is a really > cool > >> RedHat > >> >>>>>>>> project which is a minimalist replacement for docker. I was > >> recently > >> >>>>>>>> working at a startup where I migrated our "data mover" > appliance > >> from > >> >>>>>>>> Docker to CRI-O. Our application was able to get direct access > to > >> the > >> >>>>>>>> ethernet driver and block devices which enabled a huge > performance > >> boost > >> >>>>>>>> but we were also able to run containers produced by docker > without > >> >>>>>>>> modification. > > >> >>>>>>>> >>>> You mention that docker is "detail of one runner+vendor > >> corrupting all > >> >>>>>>>> the project and adding complexity and work to everyone". It > sounds > >> like you > >> >>>>>>>> have a specific example you'd like to share? Is there a runner > >> that is > >> >>>>>>>> unable to move to portability because of docker? > > > >> >>>>>>>> >>> IBM one for instance, some custom ones like an hazelcast > based > >> one, > >> >>>>>>>> etc... More generally any runner developped outside beam itself > - > >> even if > >> >>>>>>>> we take a snapshot today, most of beam's ones have the same > pitall. > > >> >>>>>>>> >>> Note: i never said docker was a bad techno or so. Let me > try > >> to clarify. > > >> >>>>>>>> >>> Main issue is that you enforce docker usage which is still > >> trendy. It > >> >>>>>>>> is like scla which was promishing to kill java, check what it > does > >> today... > >> >>>>>>>> >>> It starts to be tooled but it is also very impacting on the > >> deployment > >> >>>>>>>> side and for a good number of beam users who deploy it outside > the > >> cloud it > >> >>>>>>>> is an issue. > >> >>>>>>>> >>> Keep in mind beam is embeddable by design, it is not a > runner > >> >>>>>>>> environment and with the docker choice it imposes some > environment > >> which is > >> >>>>>>>> inconsistent with beam design itself and this is where this > choice > >> blocks. > > > > >> >>>>>>>> >>>> Andrew > > >> >>>>>>>> >>>> On Fri, May 4, 2018 at 4:32 PM Henning Rohde < > >> hero...@google.com> > >> >>>>>>>> wrote: > > >> >>>>>>>> >>>>> Romain, > > >> >>>>>>>> >>>>> Docker, unlike selinux, solves a great number of tangible > >> problems > >> >>>>>>>> for us with IMO a relatively small tax. It does not have to be > the > >> only > >> >>>>>>>> way. Some of the concerns you bring up along with possibilities > >> were also > >> >>>>>>>> discussed here: > >> >>>>>>>> https://s.apache.org/beam-fn-api-container-contract. > >> I > >> >>>>>>>> encourage you to take a look. > > >> >>>>>>>> >>>>> Thanks, > >> >>>>>>>> >>>>> Henning > > > >> >>>>>>>> >>>>> On Fri, May 4, 2018 at 3:18 PM Romain Manni-Bucau < > >> >>>>>>>> rmannibu...@gmail.com> wrote: > > > > >> >>>>>>>> >>>>>> Le 4 mai 2018 21:31, "Henning Rohde" < > hero...@google.com> > a > >> écrit : > > >> >>>>>>>> >>>>>> I disagree with the characterization of docker and the > >> implications > >> >>>>>>>> made towards portability. Graal looks like a neat project (and > I > >> never > >> >>>>>>>> thought I would live to see the phrase "Practical Partial > >> Evaluation" ..), > >> >>>>>>>> but it doesn't address the needs of portability. In addition to > >> Luke's > >> >>>>>>>> examples, Go and most other languages don't work on it either. > >> Docker > >> >>>>>>>> containers also address packaging, OS dependencies, conflicting > >> versions > >> >>>>>>>> and distribution aspects in addition to truly universal > language > >> support. > > > >> >>>>>>>> >>>>>> This is wrong, docker also has its conflicts, is not > >> universal > >> >>>>>>>> (fails on windows and mac easily - as host or not, cloud > vendors > >> put layers > >> >>>>>>>> limiting or corrupting it, and it is an infra constraint > imposed > >> and a > >> >>>>>>>> vendor locking not welcomed in beam IMHO). > > >> >>>>>>>> >>>>>> This is my main concern. All the work done looks like an > >> >>>>>>>> implemzntation detail of one runner+vendor corrupting all the > >> project and > >> >>>>>>>> adding complexity and work to everyone instead of keeping it > >> localised > >> >>>>>>>> (technically it is possible). > > >> >>>>>>>> >>>>>> Would you accept i enforce you to use selinux? Using > docker > >> is the > >> >>>>>>>> same kind of constraint. > > > >> >>>>>>>> >>>>>> That said, it's entirely fine for some runners to use > >> Jython, Graal, > >> >>>>>>>> etc to provide a specialized offering similar to the direct > >> runners, but it > >> >>>>>>>> would be disjoint from portability IMO. > > >> >>>>>>>> >>>>>> On Fri, May 4, 2018 at 10:14 AM Romain Manni-Bucau < > >> >>>>>>>> rmannibu...@gmail.com> wrote: > > > > >> >>>>>>>> >>>>>>> Le 4 mai 2018 17:55, "Lukasz Cwik" <lc...@google.com> > a > >> écrit : > > >> >>>>>>>> >>>>>>> I did take a look at Graal a while back when thinking > >> about how > >> >>>>>>>> execution environments could be defined, my concerns were > related > >> to it not > >> >>>>>>>> supporting all of the features of a language. > >> >>>>>>>> >>>>>>> For example, its typical for Python to load and call > native > >> >>>>>>>> libraries and Graal can only execute C/C++ code that has been > >> compiled to > >> >>>>>>>> LLVM. > >> >>>>>>>> >>>>>>> Also, a good amount of people interested in using ML > >> libraries will > >> >>>>>>>> want access to GPUs to improve performance which I believe that > >> Graal can't > >> >>>>>>>> support. > > >> >>>>>>>> >>>>>>> It can be a very useful way to run simple lamda > functions > >> written > >> >>>>>>>> in some language directly without needing to use a docker > >> environment but > >> >>>>>>>> you could probably use something even lighter weight then Graal > >> that is > >> >>>>>>>> language specific like Jython. > > > > >> >>>>>>>> >>>>>>> Right, the jsr223 impl works very well but you can also > >> have a perf > >> >>>>>>>> boost using native (like v8 java binding for js for instance). > It > >> is way > >> >>>>>>>> more efficient than docker most of the time and not code > intrusive > >> at all > >> >>>>>>>> in runners so likely more adoption-able and maintainable. That > >> said all is > >> >>>>>>>> doable behind the jsr223 so maybe not a big deal in terms of > api. > >> We just > >> >>>>>>>> need to ensure portability work stay clean and actually > portable > >> and doesnt > >> >>>>>>>> impact runners as poc done until today did. > > >> >>>>>>>> >>>>>>> Works for me. > > > >> >>>>>>>> >>>>>>> On Thu, May 3, 2018 at 10:05 PM Romain Manni-Bucau < > >> >>>>>>>> rmannibu...@gmail.com> wrote: > > >> >>>>>>>> >>>>>>>> Hi guys > > >> >>>>>>>> >>>>>>>> Since some time there are efforts to have a language > >> portable > >> >>>>>>>> support in beam but I cant really find a case it "works" being > >> based on > >> >>>>>>>> docker except for some vendor specific infra. > > >> >>>>>>>> >>>>>>>> Current solution: > > >> >>>>>>>> >>>>>>>> 1. Is runner intrusive (which is bad for beam and > prevents > >> >>>>>>>> adoption of big data vendors) > >> >>>>>>>> >>>>>>>> 2. Based on docker (which assumed a runtime > environment > >> and is > >> >>>>>>>> very ops/infra intrusive and likely too $$ quite often for what > it > >> brings) > > >> >>>>>>>> >>>>>>>> Did anyone had a look to graal which seems a way to > make > >> the > >> >>>>>>>> feature doable in a lighter manner and optimized compared to > >> default jsr223 > >> >>>>>>>> impls? >
