coderzc commented on code in PR #23608: URL: https://github.com/apache/pulsar/pull/23608#discussion_r1868627213
########## pip/pip-394.md: ########## @@ -0,0 +1,105 @@ +# PIP-394: Add two interfaces `CursorMetadataSerializerProvider` and `CursorMetadataDeSerializerProvider` to support newer of customized cursor metadata serializations + +# Background knowledge + +**1. What does cursor metadata contain** + +- cursor properties. +- entry id that indicates the latest persist cursor metadata into. +- information of individual acknowledged messages, we call it `individualDeletedMessages`. +- information of individual acknowledged batched messages, we call it `batchedEntryDeletionIndexInfo`. + +**2. The improvements we did for the persistence for cursor metadata** +- https://github.com/apache/pulsar/pull/758: skip to information that over the max limitation of max ranges to persist. +- https://github.com/apache/pulsar/issues/14529: compress the info when persisting. +- https://github.com/apache/pulsar/pull/9292: add a new compression strategy: change Range Objects to `long[]`. + +# Motivation + +**Issue-1: Compatible of improvements** + +- The third improvement was contributed with `release:4.0`, which is a new LTS version. + - It changed the default implementation of serialization that contains https://github.com/apache/pulsar/pull/9292. +- Users can not rollback to `3.0.x` once upgraded to `4.0.x` because `release:3.0.x` does not contain the deserialization that introduced by https://github.com/apache/pulsar/pull/9292. + +**Issue-2: Frequently Young GC relates to the cursor metadata persistence if there are too many active subscriptions in a broker, even if we did so many improvements** + +`individualDeletedMessages` and `batchedEntryDeletionIndexInfo` often is the largest attributes of the metadata. They are serialized to a proto data when being persisted. But we can not recycle the object which typed proto due to it is immutable. + + + + +# Goals + +- Guarantee compatability for rollback from `4.0.x` to `3.0.x`. + - This PIP will be cherry-picked into `branch-3.0` and `branch-3.3`. +- Support customized cursor metadata serializer to improve the issues users encountered, such as **Issue-1** in the Motivation. + +# High Level Design + +### Design + +- We call the serialization that implemented before `4.0.0` `V1`, and call after the https://github.com/apache/pulsar/pull/9292 `v2`. +- Add all version of serialization into `branch-3.0`. + - Set the default value of `3.0.x` is `V1`, which is the same as the current status. + - Set the default value of `4.0.x` is `V1`, which is the same as the current status. Review Comment: 4.0.0 default serialization is V2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
