Though we all certainly have our biases, I think it's fair to say that all of these systems are constantly innovating, borrowing ideas from one another, and have their strengths and weaknesses. I wouldn't say one is, or will always be, in front of or behind another.
Take, as the given example Spark Structured Streaming. Of course the API itself is spark-specific, but it borrows heavily (among other things) on ideas that Beam itself pioneered long before Spark 2.0, specifically the unification of batch and streaming processing into a single API, and the event-time based windowing (triggering) model for consistently and correctly handling distributed, out-of-order data streams. Of course there are also operational differences. Spark, for example, is very tied to the micro-batch style of execution whereas Flink is fundamentally very continuous, and Beam delegates to the underlying runner. It is certainly Beam's goal to keep overhead minimal, and one of the primary selling points is the flexibility of portability (of both the execution runtime and the SDK) as your needs change. - Robert On Tue, Apr 30, 2019 at 5:29 AM <[email protected]> wrote: > > Ofcourse! I suspect beam will always be one or two step backwards to the new > functionality that is available or yet to come. > > For example: Spark Structured Streaming is still not available, no CEP apis > yet and much more. > > Sent from my iPhone > > On Apr 30, 2019, at 12:11 AM, Pankaj Chand <[email protected]> wrote: > > Will Beam add any overhead or lack certain API/functions available in > Spark/Flink?
