je-ik commented on PR #29792:
URL: https://github.com/apache/beam/pull/29792#issuecomment-1959987633
I think there is no particular benefit, at least given the current
implementation. On the other hand, before we place a hard limit, we should know
it is really needed, that it is not just a manifestation of another issue.
From my understanding - if reader gets empty partitions, the particular
split should not connect to Kafka, or contribute to checkpoint size, it should
actually only emit MAX_WATERMARK (which means that there will be no more data
from this split) and sleep forever (or terminate after
`shutdownSourcesAfterIdleMs` milliseconds). Is this not working in your case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
