Jason918 opened a new issue #13544:
URL: https://github.com/apache/pulsar/issues/13544
<!---
Instructions for creating a PIP using this issue template:
1. The author(s) of the proposal will create a GitHub issue ticket using
this template.
(Optionally, it can be helpful to send a note discussing the proposal to
[email protected] mailing list before submitting this GitHub issue.
This discussion can
help developers gauge interest in the proposed changes before
formalizing the proposal.)
2. The author(s) will send a note to the [email protected] mailing list
to start the discussion, using subject prefix `[PIP] xxx`. To determine
the appropriate PIP
number `xxx`, inspect the mailing list
(https://lists.apache.org/[email protected])
for the most recent PIP. Add 1 to that PIP's number to get your PIP's
number.
3. Based on the discussion and feedback, some changes might be applied by
the author(s) to the text of the proposal.
4. Once some consensus is reached, there will be a vote to formally approve
the proposal. The vote will be held on the [email protected]
mailing list. Everyone
is welcome to vote on the proposal, though it will considered to be
binding
only the vote of PMC members. It will be required to have a lazy
majority of
at least 3 binding +1s votes. The vote should stay open for at least 48
hours.
5. When the vote is closed, if the outcome is positive, the state of the
proposal is updated and the Pull Requests associated with this proposal
can
start to get merged into the master branch.
-->
## Motivation
Currently, chunk messages producing fails if topic level maxMessageSize is
set [1]. The root cause of this issue is because chunk message is using broker
level maxMessageSize as chunk size. And topic level maxMessageSize is always <=
broker level maxMessageSize. So once it is set, the on-going chunk message
producing fails.
## Goal
Resolve topic level maxMessageSize compatibility issue with chunking
messages.
## Implementation
Current best solution would be just skipping topic level maxMessageSize
check in
org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
Topic level maxMessageSize is introduced in [2], for the purpose of "easier
to plan resource quotas for client allocation". And IMO this change will not
bring further complex into this.
## Reject Alternatives
Add a client side topic level maxMessageSize and keep it synced with broker.
Required changes:
- [client] Add a new field
org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client
side topic level maxMessageSize.
- [PulsarApi.proto] Add a new field maxMessageSize in the
CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
- [PulsarApi.proto] Add a new Command like
CommandUpdateClientPolicy{producerId, maxMessageSize} to update
ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
Further more, some other data consistency issues need be handled very
carefully when maxMessageSize is updated.
This alternative is complex but can also solve other topic level
maxMessageSize issue [3] when batching is enabled (non-batching case is solved
with PR [4]).
[1] https://github.com/apache/pulsar/issues/13360
[2] https://github.com/apache/pulsar/pull/8732
[3] https://github.com/apache/pulsar/issues/12958
[4] https://github.com/apache/pulsar/pull/13147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
