To my knowledge, no one has discussed an extension mechanism for the direct runner but the difficulty is in how to get extensions to interact with the internals of the direct runner cleanly. Note that the direct runner currently accepts a set of flags which enable/disable validation and control how it runs like "--enforceImmutability": https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectOptions.java#L49 Would it be easier to just add more flags which control how the direct runner works?
As for having a direct runner using portability to be able to execute Python / Go / Java SDKs, you should look at https://issues.apache.org/jira/browse/BEAM-2899 On Fri, Mar 2, 2018 at 12:53 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi guys, > > wonder if you discussed or thought to break down what is called today the > direct runner in an embedded runner which would be modular an extensible. > > What I have in mind is the following: > > 1. have a strong embedded runner implementing the whole beam API but > limited to a single JVM > 2. keep a string test oriented runner (what we call direct runner today) > > The overall design would be to ensure 1 and 2 share the common code and > avoid to do yet another runner. This means several extension points should > be defined to: > > 1. add the serialization validation > 2. add the portability validation > 3. add the execution randomization > > I didn't think yet to what would be the execution points (can just be > replacements probably or specific extension points which would be less > elegant but it enables to reach the same goal). > > The base runner (let's call it "EmbeddedRunner" to name it here) would > have its EmbeddedRunnerOptions which would have a --modules options to > activate all potential extension points (in > META-INF/org/apache/beam/embeddedrunner/extensions/xxx > xxx being the extension name to use in --modules for instance). > > This would enable to have an embedded runner more usable for light/small > but production oriented environments for users, would also start to align > the work done for the portability (thinking to recent python enhancements > in runners) without loosing the strong validation done in tests or preprod > envs. > > Was it already mentionned/thought? If not, wdyt? > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> >
