BewareMyPower commented on PR #24439:
URL: https://github.com/apache/pulsar/pull/24439#issuecomment-2994708222
> Actually, Pulsar only needs to retain valid message data in a compacted
entry, but it retains all compacted messages with "empty header" and "empty
payload".
Yes. Unfortunately, it's limited by client side's logic that the default
logic to parse the payload buffer looks like:
```java
int batchSize = msgMetadata.getNumMessagesInBatch();
for (int i = 0; i < batchSize; i++) {
int batchIndex = i;
final var singleMessageMetadata = parse(payload);
if (singleMessageMetadata.isCompactedOut()) {
break;
}
// Create a message, whose batch index is i, from the payload buffer
}
```
> Add a flag for the valid compacted position can avoid deserializing and
uncompressing the entire batch messages metadata header.
This makes sense to me. Adding a new field to `MessageMetadata` might be
more simple. But we need to take care of the compatibility issue. We can
simplify the payload format when this field is set. Let me try a demo first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
