lostluck commented on PR #27618: URL: https://github.com/apache/beam/pull/27618#issuecomment-1651970798
Honestly I don't think I'll be much help here either. "Wrapping" an existing runner is different from "making it work from scratch" like I've done with prism. I don't know anything about dask so I can only describe what's needed in abstract. If Dask (or this wrapper) is able to track and propagate watermarks down the pipeline, then windowing is a straightforward triggering of applicable aggregations. Otherwise, for prism, i found it impossible to map Stream Execution to a simple batch execution loop since the watermarks weren't able to behave right. So prism has an element manager that tracks available data and watermarks for stages. So, the question is then: does this dask wrapper track watermarks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
