Hi Zhixin, I would love to help you out with this.
One of the best ways to test your runner is to enable the "RunnableOnService" test suite in the core SDK. Here is an example of the configuration for the Flink runner: https://github.com/apache/incubator-beam/blob/master/runners/flink/runner/pom.xml#L49 Another good source of tests is the integration tests which makes sure your runner can run all of our user-facing examples. The configuration for this lives here: https://github.com/apache/incubator-beam/blob/master/examples/java/pom.xml#L133 You said you are almost done, but just because you might find some issues when you start running these tests, here are some pointers to other references. The standard abstract description of the model is still the Dataflow Model paper from a couple years ago: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf . And finally, we don't have extensive documentation, but best current reference for what transforms are primitive (and why) for a runner author is probably the Beam Runner API proposal at https://s.apache.org/beam-runner-api. The implementation (moving Beam to the ideal) is still under development, but you may be helped by the sections "Primitive Transforms" at https://s.apache.org/beam-runner-api#heading=h.tt55lhd3k6by and "What does a runner author need to do?" at https://s.apache.org/beam-runner-api #heading=h.cdbhozvw83un. I hope this helps. And, of course, if there are more details you can share then we can talk about specifics. Kenn On Mon, Nov 7, 2016 at 7:50 PM 李劲松(之信) <[email protected]> wrote: > Hi there, > > I'm working on the beam integration (= a new runner, mainly for streaming; > almost done) for an internal system at Alibaba, targeted for production use. > I'm wondering if you could give me some advice on how to test/verify such an > implementation. Thank you! > > Best, > Zhixin
