If all engines were identical, having a shared optimizer would be useful. Having a proxy runner that performance optimizations before submission to an actual engine-specific runner has downsides in both directions:
- obscures the ability of engine-specific runners to optimize the Beam primitives because they only receive post-optimized graph - has to be extremely conservative in its optimizations because it does not know about the semantics of the underlying engine Building it as libraries let's engine-specific runners do what is best for their engine, while still maximizing reuse. Kenn On Thu, May 17, 2018 at 11:43 AM Robert Burke <rob...@frantil.com> wrote: > The approach you're looking for sounds like the user's Runner of Choice, > would use a user side version of the runner core, without changing the > Runner of Choice? > > So a user would update their version of the SDK, and the runner would have > to pull the core component from the user pipeline? > > That sounds like it increases pipeline size and decreases pipeline > portability, especially for pipelines that are not in the same language as > the runner-core, such as for Python and Go. > > It's not clear to me what runners would be doing in that scenario either. > Do you have a proposal about where the interface boundaries would be? > > On Wed, May 16, 2018, 10:05 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> The runner core doesnt fully align on that or rephrased more accurately, >> it doesnt go as far as it could for me. Having to call it, is still an >> issue since it requires a runner update instead of getting the new feature >> for free. The next step sounds to be *one* runner where implementations >> plug their translations probably. It would reverse the current pattern and >> prepare beam for the future. One good example of such implementation is the >> sdf which can "just" reuse dofn primitives to wire its support through >> runners. >> >> Le jeu. 17 mai 2018 02:01, Jesse Anderson <je...@bigdatainstitute.io> a >> écrit : >> >>> This -> "I'd like that each time you think that you ask yourself "does >>> it need?"." >>> >>> On Wed, May 16, 2018 at 4:53 PM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> Thanks for your email, Romain. It helps understand your goals and where >>>> you're coming from. I'd also like to see a thinner core, and agree it's >>>> beneficial to reduce dependencies where possible, especially when >>>> supporting the usecase where the pipeline is constructed in an >>>> environment >>>> other than an end-user's main. >>>> >>>> It seems a lot of the portability work, despite being on the surface >>>> driven >>>> by multi-language, aligns well with many of these goals. For example, >>>> all >>>> the work going on in runners-core to provide a rich library that all >>>> (Java, >>>> and perhaps non-Java) runners can leverage to do DAG preprocessing >>>> (fusion, >>>> combiner lifting, ...) and handle the low-level details of managing >>>> worker >>>> subprocesses. As you state, the more we can put into these libraries, >>>> the >>>> more all runners can get "for free" by interacting with them, while >>>> still >>>> providing the flexibility to adapt to their differing models and >>>> strengths. >>>> >>>> Getting this right is, for me at least, one of the highest priorities >>>> for >>>> Beam. >>>> >>>> - Robert >>>> On Wed, May 16, 2018 at 11:51 AM Kenneth Knowles <k...@google.com> >>>> wrote: >>>> >>>> > Hi Romain, >>>> >>>> > This gives a clear view of your perspective. I also recommend you ask >>>> around to those who have been working on Beam and big data processing >>>> for a >>>> long time to learn more about their perspective. >>>> >>>> > Your "Beam Analysis" is pretty accurate about what we've been trying >>>> to >>>> build. I would say (a) & (b) as "any language on any runner" and (c) is >>>> our >>>> plan of how to do it: define primitives which are fundamental to >>>> parallel >>>> processing and formalize a language-independent representation, with >>>> adapters for each language and data processing engine. >>>> >>>> > Of course anyone in the community may have their own particular goal. >>>> We >>>> don't control what they work on, and we are grateful for their efforts. >>>> >>>> > Technically, there is plenty to agree with. I think as you learn about >>>> Beam you will find that many of your suggestions are already handled in >>>> some way. You may also continue to learn sometimes about the specific >>>> reasons things are done in a different way than you expected. These >>>> should >>>> help you find how to build what you want to build. >>>> >>>> > Kenn >>>> >>>> > On Wed, May 16, 2018 at 1:14 AM Romain Manni-Bucau < >>>> rmannibu...@gmail.com> >>>> wrote: >>>> >>>> >> Hi guys, >>>> >>>> >> Since it is not the first time we have a thread where we end up not >>>> understanding each other, I'd like to take this as an opportunity to >>>> clarify what i'm looking for, in a more formal way. This assumes our >>>> misunderstandings come from the fact I mainly tried to fix issues one by >>>> ones, instead of painting the big picture I'm getting after. (My >>>> rational >>>> was I was not able to invest more time in that but I start to think it >>>> was >>>> not a good chocie). I really hope it helps. >>>> >>>> >> 1. Beam analysis >>>> >>>> >> Beam has three main goals: >>>> >>>> >> a. Being a portable API accross runners (I also call them >>>> "implementations" by opposition of "api") >>>> >> b. Bringing some interoperability between languages and therefore >>>> users >>>> >> c. Provide primitives (groupby for instance), I/O and generic >>>> processing >>>> items >>>> >>>> >> Indeed it doesn't cover all beam's features but, high level, it is >>>> what >>>> it brings. >>>> >>>> >> In terms of advantages and why choosing beam instead of spark, for >>>> instance, the benefit is mainly to not be vendor locked on one side and >>>> to >>>> enable more users on the other side (you note that point c is just >>>> catching >>>> up on vendors ecosystems with these statements). >>>> >>>> >> 2. Portable API accross environments >>>> >>>> >> It is key, here, to keep in mind beam is not an environment or a >>>> runner. >>>> It is by design, a library *embedded* in other environment. >>>> >>>> >> a. This means that Beam must keep its stack as clean as possible. If >>>> it >>>> is still ambiguous: beam must be dependency free. >>>> >>>> >> Until now the workaround has been to shade dependencies. This is not >>>> a >>>> solution since it leads to big jobs of hundreds of mega which prevents >>>> to >>>> scale since we deploy from the network. It makes all deployments, >>>> managements, and storage a pain on ops side. The other pitfall of shades >>>> (or shadowing since we are on gradle now) is that it completely breaks >>>> any >>>> company tooling and prevent vulnerability scanning or dependency >>>> upgrades - >>>> not handled by dev team - to work correctly. This is a major issue for >>>> any >>>> software targetting some professional level which should not be >>>> underestimated. >>>> >>>> >> From that point we can get scared but with Java 8 there is no real >>>> point >>>> having a tons of dependencies for the sdk core - this is for java but >>>> should be true for most languages since beam requirements are light >>>> here. >>>> >>>> >> However it can also require to rethink the sdk core modularity: why >>>> is >>>> there some IO here? Do we need a big fat sdk core? >>>> >>>> >> b. API or "put it all"? >>>> >>>> >> Current API is in sdk-core but actually it prevents a modular >>>> development since there are primitives and some IO in the core. What >>>> would >>>> be sane is to extract the actual API from the core and get a beam-api. >>>> This >>>> way we match all kind of user consumes: >>>> >>>> >> - IO developers (they only need the SDF) >>>> >> - pipeline writers (they only need the pipeline + IO) >>>> >> - etc... >>>> >>>> >> To make it an API it requires some changes but nothing crazy probably >>>> and it would make beam more consumable and potentially reusable in other >>>> environments. >>>> >>>> >> I'll not detail the API points here since it is not the goal (think I >>>> tracked most of them in >>>> https://gist.github.com/rmannibucau/ab7543c23b6f57af921d98639fbcd436 >>>> if you >>>> are interested) >>>> >>>> >> c. Environment is not only about jars >>>> >>>> >> Beam has two main execution environments: >>>> >>>> >> - the "pipeline.run" one >>>> >> - the pipeline execution (runner) >>>> >>>> >> The last one is quite known and already has some challenges: >>>> >>>> >> - can be a main execution so nothing crazy to manage >>>> >> - can use subclassloaders to execute jobs, scale and isolate jobs >>>> >> - etc... (we can think to an OSGi flavor for instance) >>>> >>>> >> The first one is way more challenging since you must match: >>>> >>>> >> - flat mains >>>> >> - JavaEE containers >>>> >> - OSGi containers >>>> >> - custom weird environments (spring boot jar launcher) >>>> >> - ... >>>> >>>> >> This all lead to two very key consequences and programming rule >>>> respect: >>>> >>>> >> - lifecycle: any component must ensure its lifecycle is very well >>>> respected (we must avoid "JVM will clean up anyway" kind of thinking) >>>> >> - no blind cache or static abuse, this must fit *all* environments >>>> (pipelineoptionsfacctory is a good example of that) >>>> >>>> >> 3. Make it hurtless for integrators/community >>>> >>>> >> Beam's success is bound to the fact runners exist. A concern which is >>>> quite important is that beam keeps adding features and say "runners will >>>> implement them". I'd like that each time you think that you ask yourself >>>> "does it need?". >>>> >>>> >> I'll take two examples: >>>> >>>> >> - the language portable support: there is no need to do it in all >>>> runners, you can have a generic runner delegating to the right >>>> implementation@runner the tasks and therefore, adding language >>>> portability >>>> feature, you support OOTB all existing runners without impacting them >>>> >> - the metrics pusher: this one got some discussion and lead to a >>>> polling >>>> implementation which doesn't work in all runners not having a waiting >>>> "driver" (hazelcast, spark in client mode etc...). Now it is going to be >>>> added to the portable API if I got it right...if you think about it, you >>>> can just instrument the pipeline by modifying the DAG before >>>> translating it >>>> and therefore work on all runners for free as well. >>>> >>>> >> These two simple examples show that the work should probably be done >>>> on >>>> adding DAG preprocessors (sorted) and runner as something enrichable, >>>> rather than with ad-hoc solutions for each feature. >>>> >>>> >> 4. Be more reactive >>>> >>>> >> If you check I/O, most of them can support asynchronous handling. The >>>> gain is to be aligned on the actual I/O and not only be asynchronous to >>>> starts a new thread. Using that allows to scale way more and use more >>>> efficiently resources of the machine. >>>> >>>> >> However it has a big pitfall: the whole programming model must be >>>> reactive. Indeed, we can support a conversion from a not reactive to a >>>> reactive model implicitly for simple case (think to a DoFn multiplying >>>> by 2 >>>> an int) but the I/O should be reactive and beam should be reactive in >>>> its >>>> completion to benefit from it. >>>> >>>> >>>> >>>> >> Summary: if I try to summarize this mail which tries to share the >>>> philosophy I'm approaching beam with, more than particular issues, i'd >>>> say >>>> that I strongly think, that to be a success, Beam but embrace what it >>>> is: a >>>> portable layer on top of existing implementations. It means that it must >>>> define a clear and minimal API for each kind of usage and probably >>>> expose >>>> it by user kind (so actually N api). it must embrace the environments it >>>> runs in and assume the constraints it brings. And finally it should be >>>> less >>>> intrusive in all its layers and try to add features more transversally >>>> when >>>> possible (and it is possible in a lot of cases). If you bring features >>>> for >>>> free with new releases, everybody wins, if you announce features and no >>>> runner support it, then you loose (and loose users). >>>> >>>> >>>> >>>> >> Hope it helps, >>>> >> Romain Manni-Bucau >>>> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>> >>>
