All runners just provide translations so it is easy to build features on
top of primitives, ie basic translations, instead of requiring runners to
use the same lib which is not yet done and will likely not be done when
adding new reusable parts - keep in mind beam starts to have runners not
hosted
If all engines were identical, having a shared optimizer would be
useful. Having
a proxy runner that performance optimizations before submission to an
actual engine-specific runner has downsides in both directions:
- obscures the ability of engine-specific runners to optimize the Beam
primitives
The approach you're looking for sounds like the user's Runner of Choice,
would use a user side version of the runner core, without changing the
Runner of Choice?
So a user would update their version of the SDK, and the runner would have
to pull the core component from the user pipeline?
That soun
The runner core doesnt fully align on that or rephrased more accurately, it
doesnt go as far as it could for me. Having to call it, is still an issue
since it requires a runner update instead of getting the new feature for
free. The next step sounds to be *one* runner where implementations plug
the
This -> "I'd like that each time you think that you ask yourself "does it
need?"."
On Wed, May 16, 2018 at 4:53 PM Robert Bradshaw wrote:
> Thanks for your email, Romain. It helps understand your goals and where
> you're coming from. I'd also like to see a thinner core, and agree it's
> benefici
Thanks for your email, Romain. It helps understand your goals and where
you're coming from. I'd also like to see a thinner core, and agree it's
beneficial to reduce dependencies where possible, especially when
supporting the usecase where the pipeline is constructed in an environment
other than an
Hi Romain,
This gives a clear view of your perspective. I also recommend you ask
around to those who have been working on Beam and big data processing for a
long time to learn more about their perspective.
Your "Beam Analysis" is pretty accurate about what we've been trying to
build. I would say
Hi guys,
Since it is not the first time we have a thread where we end up not
understanding each other, I'd like to take this as an opportunity to
clarify what i'm looking for, in a more formal way. This assumes our
misunderstandings come from the fact I mainly tried to fix issues one by
ones, inst