nodece opened a new issue #14529:
URL: https://github.com/apache/pulsar/issues/14529
<!---
Instructions for creating a PIP using this issue template:
1. The author(s) of the proposal will create a GitHub issue ticket using
this template.
(Optionally, it can be helpful to send a note discussing the proposal to
[email protected] mailing list before submitting this GitHub issue.
This discussion can
help developers gauge interest in the proposed changes before
formalizing the proposal.)
2. The author(s) will send a note to the [email protected] mailing list
to start the discussion, using subject prefix `[PIP] xxx`. To determine
the appropriate PIP
number `xxx`, inspect the mailing list
(https://lists.apache.org/[email protected])
for the most recent PIP. Add 1 to that PIP's number to get your PIP's
number.
3. Based on the discussion and feedback, some changes might be applied by
the author(s) to the text of the proposal.
4. Once some consensus is reached, there will be a vote to formally approve
the proposal. The vote will be held on the [email protected]
mailing list. Everyone
is welcome to vote on the proposal, though it will considered to be
binding
only the vote of PMC members. It will be required to have a lazy
majority of
at least 3 binding +1s votes. The vote should stay open for at least 48
hours.
5. When the vote is closed, if the outcome is positive, the state of the
proposal is updated and the Pull Requests associated with this proposal
can
start to get merged into the master branch.
-->
## Motivation
The cursor data is managed by ZooKeeper/etcd metadata store. When cursor
data becomes more and more, the data size will increase and will take a lot of
time to pull the data. Therefore, it is necessary to add compression for the
cursor, which can reduce the size of data and reduce the time of pulling data.
## Goal
Support use the LZ4/ZLIB/ZSTD/SNAPPY to compress the ManagedCursorInfo.
## Implementation
- Cursor compression format
```
[MAGIC_NUMBER] + [METADATA_SIZE] + [METADATA_PAYLOAD] +
[MANAGED_CURSOR_INFO_PAYLOAD]
```
- MAGIC_NUMBER
Ox4779
- METADATA
Add a named `ManagedCursorInfoMetadata` message to `MLDataFormats.proto`:
```
message ManagedCursorInfoMetadata {
required CompressionType compressionType = 1;
required int32 uncompressedSize = 2;
}
```
Currently, these compressions have been supported, we only need to deal with
compression and decompression of the `ManagedCursorInfo` data:
- Get CursorInfo from the metadata store
We will check the cursor data header, if it is compressed, we will parse
the bytes data by compressed format, otherwise by the original way.
- Add/Update CursorInfo to the metadata store
The default is to use compression if the compression type is specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
