Eliaaazzz commented on issue #19137: URL: https://github.com/apache/beam/issues/19137#issuecomment-4055866337
For additional context, I’ve been exploring this in parallel as part of my GSoC preparation, and my main takeaway so far is that this problem extends beyond foundational API definitions. In a separate public PoC (https://github.com/Eliaaazzz/beam/tree/gsoc/poc), I’ve been validating an end-to-end SDF-style UnboundedSource path, including restriction/tracker/coder handling, checkpoint persistence and finalization, watermark restore/progression, deferred resumption, and a standalone reader transform. The UnboundedSource PoC currently has 49 passing tests. What became clear from implementing it is that the core difficulty here is not the class surface itself, but the execution contract and runtime semantics: checkpoint-based splitting, primary/residual representation, watermark behavior, idle resume behavior, Read(UnboundedSource) integration, and alignment with Beam’s existing Java/Python model all need to be pinned down carefully. I’m currently developing this as the main direction of my GSoC proposal, so I wanted to share the PoC here because I think those end-to-end semantic/runtime requirements are central to the overall design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
