tkaymak commented on PR #39576: URL: https://github.com/apache/beam/pull/39576#issuecomment-5341463035
Hi @Eliaaazzz, happy to share where this is heading. The thinking behind the POC: Spark 4 added transformWithState, and it turned out flexible enough to host the Beam streaming model directly. The POC proves the core claim, two chained Beam stateful operators run inside a single Structured Streaming query with a watermark that propagates correctly between them. So yes, this is the intended direction, and I would not build new work on the DStream path. Once there is agreement on the approach, the plan is to split this branch into five smaller PRs: the dispatch seam and options, the Kryo registrations, the DataSourceV2 unbounded source, the state and timer bridge, and the translators with the end to end tests. Concrete places where help would be useful right now: 1. Tests. The shared base has four tests marked "TODO: Reactivate with streaming." plus SimpleSourceTest.testUnboundedSource. They cannot be enabled in place because Spark 3 compiles the same test tree and still correctly rejects streaming. Moving them into the Spark 4 override tree is a nice contained task. 2. The PAssert story. No source emits a final infinity watermark, so panes never finalize and PAssert does not work on this path. All streaming tests currently assert against a static collector. Reviewers will ask about this, so ideas or a prototype here would be valuable. 3. Checkpoint recovery. Readers currently resume from a per JVM cache, so restarting from a checkpoint written by an earlier run is not supported. This is the biggest gap between the POC and something shippable. Streaming side inputs, your original interest, are deliberately out of scope for the POC, but they are a natural follow up phase once this foundation is agreed. Session windows and full trigger support are planned as later phases too. If you want to start somewhere, I would suggest number 1, it is contained and it teaches you the test setup you would need for anything bigger. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
