Caideyipi opened a new pull request, #18471:
URL: https://github.com/apache/iotdb/pull/18471
## Description
### Root cause
The subscription brokers polled events before enforcing the cumulative
response payload limit. When several individually valid events were selected
for one response, the receiver could discover only during serialization that
the cumulative payload exceeded the poll budget. It then nacked the event.
After 10 identical retries, poison-message handling force-acked the event and
permanently skipped its rows.
This matches regression test case 1392 exactly: two skipped writer_B events,
each containing 10 rows, account for the 20-row gap between benchmark/IoTDB
count (42,423,170) and consumed count (42,423,150).
The failing event itself was about 30 MB, not 68 MB. The reported 68,135,206
bytes was the cumulative response size after adding it. The effective server
threshold was 60,397,977 bytes, which is 90% of the consumer's 64 MiB poll
budget. Therefore the write path's payload limit was not violated; whether a
writer_B event succeeded depended on how much data had already been selected
for that poll response.
### Fix
- Check the cumulative payload budget before adding an event in both
consensus and pipe subscription brokers.
- Return an event that does not fit to its prefetch queue atomically,
without calling `nack()` or increasing `nackCount`.
- Apply the same budget enforcement across multiple brokers in
`SubscriptionBrokerAgent`.
- Keep receiver-side overflow handling as a race-condition fallback: requeue
cumulative-overflow events and nack only when requeue is not applicable or
fails.
- Preserve existing behavior for a single event that is larger than the
complete response budget, avoiding an infinite requeue loop.
### Verification
- `mvn spotless:apply -pl iotdb-core/datanode`
- `mvn -o test-compile -pl iotdb-core/datanode -DskipTests`
- `ConsensusSubscriptionBrokerPayloadLimitTest`: 1 test, 0 failures/errors
- `SubscriptionBrokerAgentPayloadLimitTest`: 1 test, 0 failures/errors
- `ConsensusPrefetchingQueueTest`: 27 tests, 0 failures/errors
- `git diff --check`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added Javadocs for non-trivial methods.
- [x] added comments explaining the intent of payload-overflow requeue
behavior.
- [x] added or updated unit tests for the new code paths.
<hr>
##### Key changed/added classes
- `SubscriptionBrokerAgent`
- `ConsensusSubscriptionBroker`
- `SubscriptionBroker`
- `ConsensusPrefetchingQueue`
- `SubscriptionPrefetchingQueue`
- `SubscriptionReceiverV1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
