poorbarcode opened a new pull request, #25244: URL: https://github.com/apache/pulsar/pull/25244
### Background 1. The attributes `cursor.active` and `cursor.lastActive` are applied in different scenarios, they are not in the same fields. - `cursor.active: boolean`: whether the consumption catched up - `cursor.lastActive`: the latest time that the cursor was touched, such as consumer registation, consumer unregistation, reset cursors. --- 2. The usages of `cursor.lastActive` - Relates to the configuration `broker.conf -> subscriptionExpirationTimeMinutes`. Let Broker knows whether the cursor has been idle for a long time, to correctly delete the inactive subscriptions. - Let users know the latest time of touching the curosr --- 3. The history of changing of `cursor.lastActive` 3-1. update `cursor.lastActive` to the current timestamp when consumers register, unregister, reset cursors. Broker persists it to metadata store when closing topics. 3-2. Pulsar supports persisting cursor state to Bookies. 3-3. https://github.com/apache/pulsar/pull/17573 removed the mechanism that recovering `cursor.lastActive` from Metadata store, to fix an issue<sup>[1]</sup> --- - **[1]**: The issue that losses messages - `topic.LAC`:`3:99` - `subscriptionExpirationTimeMinutes`: `1 day` - `consumers`: `["c1"]` - `c1` registered at `2026-02-12 00:00` - `cursor.markDeletedPosition`: `3:0` - `c1` acknowledged message `3:0`at `2026-02-12 00:01` - `cursor.lastActive` was set to `2026-02-12 00:01` and was persisted into Metadata store - `c1` do not acknowlege any messages anymore. - The broker was crashed at `2026-02-13 01:00` - The topic was reloaded up. - The broker recovered `cursor.lastActive` to `2026-02-12 00:01` - Since the time range `2026-02-12 00:01 ~ 2026-02-13 01:00` is longer than `1 day`, the subscription was deleted, but the consumer is still connected until the broker crashed. - Messages `3:0(exclusive) ~ 3:99` were lost --- ### Motivation **Issues 1**. The inactive subscription can not be deleted automaticaily if bundle rebalances frequently - `subscriptionExpirationTimeMinutes`: `1 day` - `2026-02-12 00:00`: all consumers were offline at - `2026-02-12 12:00`: rebalanced: the topic was reloaded - since https://github.com/apache/pulsar/pull/17573 removed the mechanism that persists `cursor.lastActive`, the value of `cursor.lastActive` will be reset to `system.currentMillis` - `2026-02-13 06:00`: rebalanced: the topic was reloaded - since https://github.com/apache/pulsar/pull/17573 removed the mechanism that persists `cursor.lastActive`, the value of `cursor.lastActive` will be reset to `system.currentMillis` - The subscription can never be deleted, even though it takes much longer than the threshold time. **Issue 2**: when adding the new feature that Pulsar supports persisting cursor state to Bookies, it forgot to persist `cursor.lastActive` to Bookies, which will also cause the issue that inactive subscription can not be deleted automaticaily. ### Modifications - Regarding to **Issue 2**, persist `cursor.lastActive` to Bookie also, since the field is typed `optional`, there is no compatibilty issues in multi versions - Regarding to**Issue 1**, change the mechanism as follows: - Update `cursor.lastActive` to a negative value (`v = 0 - system.currentMillis`), when all consumers are unregister. - It is a marker that means the inactive time. - Update `cursor.lastActive` when consumers register. - It is the original meanning of the field. - How to determine whether the subscription should be deleted. - `cursor.lastActive` is negative and `{current time} - Math.abs(cursor.lastActive) > {threshold}` - Persists `cursor.lastActive` when the first consumer registers, to avoid the issue that https://github.com/apache/pulsar/pull/17573 fixed. ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
