RobertIndie commented on code in PR #22651: URL: https://github.com/apache/pulsar/pull/22651#discussion_r1593816786
########## pip/pip-349.md: ########## @@ -0,0 +1,33 @@ +# PIP-349: Add additionalSystemCursorNames ignore list for ttl check + +# Background knowledge + +In Pulsar topic, we have [retention policy](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies) to control the acknowledged message lifetime. For the unacknowledged messages, we have a separate mechanism to control the message lifetime, which is called [`TTL`](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#time-to-live-ttl). The `TTL` is a time-to-live value for the message, which is controlled by `ttlDurationDefaultInSeconds`. The message will be automatically acknowledged if it is not consumed within the `TTL` value. + +# Motivation + +In Pulsar, we have two kinds of topics, system topic and normal topic. The system topics are used for internal purposes, such as transaction internal topics. The system topics are not supposed to be consumed by the users. However, the system topics are still subject to the `TTL` check. If the system topics are not consumed within the `TTL` value, the messages in the system topics will be automatically acknowledged. This is not the expected behavior for the system topics and may lead to data loss. +For normal topics, we also has two kinds of subscriptions, system subscription and normal subscription. The system subscription is used for internal purposes, such as compaction service or third-party plugins. The system subscription is not supposed to be used by the users. However, the system subscription is still subject to the `TTL` check. If the system subscription is not consumed within the `TTL` value, the messages in the system subscription will be automatically acknowledged. This is not the expected behavior for the system subscription. + +We had one PR [#21865](https://github.com/apache/pulsar/pull/21865) to filter the compaction service cursors for TTL check, but it doesn't cover other system cursors. To provide a general solution and support third-party plugin cursors not impacted by TTL, I proposed to add an additionalSystemCursorNames ignore list to filter the TTL check. + +# Goals + +## In Scope + +Add an additionalSystemCursorNames ignore list to filter the TTL check for additional system subscriptions except for compaction service subscription. The systemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. +The compaction service subscription is a system subscription and should not be impacted by TTL. To reduce the risk of data loss after enabled compaction service, we will add the compaction service subscription to the TTL ignore list by default and can't be removed. + +# Detailed Design + +## Design & Implementation Details + +Add a systemCursorNames ignore list to filter the TTL check for system subscriptions. The systemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. Review Comment: ```suggestion Add a additionalSystemCursorNames ignore list to filter the TTL check for system subscriptions. The additionalSystemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. ``` ########## pip/pip-349.md: ########## @@ -0,0 +1,33 @@ +# PIP-349: Add additionalSystemCursorNames ignore list for ttl check + +# Background knowledge + +In Pulsar topic, we have [retention policy](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies) to control the acknowledged message lifetime. For the unacknowledged messages, we have a separate mechanism to control the message lifetime, which is called [`TTL`](https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#time-to-live-ttl). The `TTL` is a time-to-live value for the message, which is controlled by `ttlDurationDefaultInSeconds`. The message will be automatically acknowledged if it is not consumed within the `TTL` value. + +# Motivation + +In Pulsar, we have two kinds of topics, system topic and normal topic. The system topics are used for internal purposes, such as transaction internal topics. The system topics are not supposed to be consumed by the users. However, the system topics are still subject to the `TTL` check. If the system topics are not consumed within the `TTL` value, the messages in the system topics will be automatically acknowledged. This is not the expected behavior for the system topics and may lead to data loss. +For normal topics, we also has two kinds of subscriptions, system subscription and normal subscription. The system subscription is used for internal purposes, such as compaction service or third-party plugins. The system subscription is not supposed to be used by the users. However, the system subscription is still subject to the `TTL` check. If the system subscription is not consumed within the `TTL` value, the messages in the system subscription will be automatically acknowledged. This is not the expected behavior for the system subscription. + +We had one PR [#21865](https://github.com/apache/pulsar/pull/21865) to filter the compaction service cursors for TTL check, but it doesn't cover other system cursors. To provide a general solution and support third-party plugin cursors not impacted by TTL, I proposed to add an additionalSystemCursorNames ignore list to filter the TTL check. + +# Goals + +## In Scope + +Add an additionalSystemCursorNames ignore list to filter the TTL check for additional system subscriptions except for compaction service subscription. The systemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. Review Comment: ```suggestion Add an additionalSystemCursorNames ignore list to filter the TTL check for additional system subscriptions except for compaction service subscription. The additionalSystemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
