MansiSingh17 commented on issue #18479:
URL: https://github.com/apache/beam/issues/18479#issuecomment-3938436847
@junaiddshaukat Hi @junaiddshaukat, thanks for the correction — I apologize
for the mix-up. My contributions are actually to the Beam Python SDK: #37672
(fix GroupBy snippet tests and re-enable skipped assertions) and #37674 (make
GCS filesystem lookup lazy to match S3 behavior). I shouldn't have referenced
TypeScript SDK PRs in my original comment.
Thank you for the detailed answers. A couple of follow-up thoughts:
- On watermark advancement: For partitions that are idle with no new records
arriving, would we still need a periodic wall-clock punctuation to trigger the
watermark check? Or does the source reader advance independently of data flow?
- On error handling: Since we have manual commit control, failed bundles
could potentially replay from the last committed offset rather than failing the
whole job. Worth thinking about early given it affects how bundle boundaries
are designed.
I'm happy to contribute to any part of this project. Is there an aspect of
the implementation where an additional contributor would be helpful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
