On Tue, Sep 11, 2018 at 7:01 PM Henning Rohde <[email protected]> wrote:
> > Empty pipelines have neither subtransforms or a spec, which is what I > don't think is useful > There's nothing preventing them form having a spec, display data, etc. They're useful because they (more) faithfully represent the user's intent, the same as all other composites the runner is not aware of. -- especially given the only usecase (which is really "nop") would create > non-timer loops in the representations. I'd rather have a well-known nop > primitive instead. Even now, for the A example, I don't think it's > unreasonable to add a (well-known) identity transform inside a normal > composite to retain the extrema at either end. It could be ignored at > runtime at no cost. > To me this feels like designing a containers library, but disallowing empty containers. Why would anyone want to use a container that's empty? Well, it turns out that's very useful and disallowing it means adding all kinds of exceptions and guards (or runtime errors) in any code using it. Having a "put this no-op element in if you want your container to be semantically empty" isn't a good solution either. Maybe it's just the mathematician in me, but disallowing this well-defined if degenerate case from the spec seems bad (especially if it's because a particular library neither handles nor drops them yet). On Fri, Sep 14, 2018 at 10:28 PM Lukasz Cwik <[email protected]> wrote: > Robert, in my mind making the shared libraries extensible for internal > usage would be B2. Exposing those native PTransforms within the community > would be B3. > > It seems as though we can achieve both A1 and B2 by making the shared > library extensible. I have this commit[1] which starts to allow the shared > libraries to be extensible and support native transformers. > >> >>>>>>> <https://github.com/apache/beam/blob/6df2ef3ec9c835097e79b4441ce47ff09a458894/model/pipeline/src/main/proto/beam_runner_api.proto#L180> >>>>>> >>>>>> Yes, this looks like a good pragmatic solution.
