syhily commented on pull request #17119: URL: https://github.com/apache/flink/pull/17119#issuecomment-913197813
> I think we have an issue with non-exclusive topic partitions as-is: > > * If we have a full restart, it makes sense to reinitialize on the enumerator. > * For partial restarts (regional failover), I don't see how we can plausibly initialize the reader: > * If we initialize on enumerator, we interfere with non-failed subtasks. > * If we initialize on the reader, we get the current issues. > * If we don't initialize, it's probably incorrect as well The issue of non-exclusive topic partitions doesn't exist. Let me clarify it. Consuming with the `Shared` or `Key_Shared` (on non-exclusive topic partitions) subscription would share the same partition among different readers. We can't use cumulative acknowledgment in these two modes, which means we have to mark the messages as read one by one. Record all the consumed messages and marked as read after successfully performing checkpoint doesn't sound like a good design. So we use [Transaction](https://pulsar.apache.org/docs/en/txn-how/#3-acknowledge-messages-with-a-transaction) to solve this problem. When we met partial restarts, the uncommitted Transaction would be abort, we would redeliver all the unacknowledged messages (the messages which don't mark read) by using `consumer.redeliverUnacknowledgedMessages()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
