Re: Beam high level directions (was "Graal instead of docker?")

All runners just provide translations so it is easy to build features on top of primitives, ie basic translations, instead of requiring runners to use the same lib which is not yet done and will likely not be done when adding new reusable parts - keep in mind beam starts to have runners not hosted

Re: Beam high level directions (was "Graal instead of docker?")

If all engines were identical, having a shared optimizer would be useful. Having a proxy runner that performance optimizations before submission to an actual engine-specific runner has downsides in both directions: - obscures the ability of engine-specific runners to optimize the Beam primitives

Re: Beam high level directions (was "Graal instead of docker?")

The approach you're looking for sounds like the user's Runner of Choice, would use a user side version of the runner core, without changing the Runner of Choice? So a user would update their version of the SDK, and the runner would have to pull the core component from the user pipeline? That

Re: Beam high level directions (was "Graal instead of docker?")

The runner core doesnt fully align on that or rephrased more accurately, it doesnt go as far as it could for me. Having to call it, is still an issue since it requires a runner update instead of getting the new feature for free. The next step sounds to be *one* runner where implementations plug

Re: Beam high level directions (was "Graal instead of docker?")

This -> "I'd like that each time you think that you ask yourself "does it need?"." On Wed, May 16, 2018 at 4:53 PM Robert Bradshaw wrote: > Thanks for your email, Romain. It helps understand your goals and where > you're coming from. I'd also like to see a thinner core, and

Re: Beam high level directions (was "Graal instead of docker?")

Thanks for your email, Romain. It helps understand your goals and where you're coming from. I'd also like to see a thinner core, and agree it's beneficial to reduce dependencies where possible, especially when supporting the usecase where the pipeline is constructed in an environment other than an

Re: Beam high level directions (was "Graal instead of docker?")

Hi Romain, This gives a clear view of your perspective. I also recommend you ask around to those who have been working on Beam and big data processing for a long time to learn more about their perspective. Your "Beam Analysis" is pretty accurate about what we've been trying to build. I would say

Beam high level directions (was "Graal instead of docker?")

Hi guys, Since it is not the first time we have a thread where we end up not understanding each other, I'd like to take this as an opportunity to clarify what i'm looking for, in a more formal way. This assumes our misunderstandings come from the fact I mainly tried to fix issues one by ones,