Hi guys,

wonder if you discussed or thought to break down what is called today the
direct runner in an embedded runner which would be modular an extensible.

What I have in mind is the following:

1. have a strong embedded runner implementing the whole beam API but
limited to a single JVM
2. keep a string test oriented runner (what we call direct runner today)

The overall design would be to ensure 1 and 2 share the common code and
avoid to do yet another runner. This means several extension points should
be defined to:

1. add the serialization validation
2. add the portability validation
3. add the execution randomization

I didn't think yet to what would be the execution points (can just be
replacements probably or specific extension points which would be less
elegant but it enables to reach the same goal).

The base runner (let's call it "EmbeddedRunner" to name it here) would have
its EmbeddedRunnerOptions which would have a --modules options to activate
all potential extension points (in
META-INF/org/apache/beam/embeddedrunner/extensions/xxx xxx being the
extension name to use in --modules for instance).

This would enable to have an embedded runner more usable for light/small
but production oriented environments for users, would also start to align
the work done for the portability (thinking to recent python enhancements
in runners) without loosing the strong validation done in tests or preprod

Was it already mentionned/thought? If not, wdyt?

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book

Reply via email to