Hi everyone, As part of the Go SDK development, I was looking at the guidelines for merging new runners and SDKs into master [1] and I think they would benefit from being updated to reflect the emerging portability framework. Specific suggestions:
(1) Both runners and SDKs should support the portability framework (to the extent the model is supported by the runner/SDK). It would be counter-productive at this time for the ecosystem to go against that effort without a compelling reason. Direct runners not included. (2) What are the minimal set of IO connectors a new SDK must support? Given the upcoming cross-language feature in the portability framework, can we rely on that to meet the requirement without implementing any native IO connectors? (3) Similarly to new runners, new SDKs should handle at least a useful subset of the model, but not necessarily the whole model (at the time of merge). A global-window-batch-only SDK targeting the portability framework, for example, could be as useful a contribution in master as a full model SDK that is supported by a direct runner only. Of course, this is not to say that SDKs should not strive to support the full model, but rather -- like Python streaming -- that it's fine to pursue that goal in master beyond a certain point. That said, I'm curious as to why this guideline for SDKs was set that specifically originally. Finally, while portability support for various features -- such as side input, cross-language I/O and the reference runner -- is still underway, what should the guidelines be? For the Go SDK specifically, if in master, it would bring the additional utility of helping test the portability framework as it's being developed. On the other hand, it can't support features that do not yet exist. What do you all think? Thanks, Henning [1] https://beam.apache.org/contribute/feature-branches/
