Caideyipi opened a new pull request, #18812: URL: https://github.com/apache/iotdb/pull/18812
## Description In a mixed large-field and numeric subscription workload, one queue retained most of a DataNode's subscription materialization budget while other queues also rejected real-time entries. The old warning called every rejection "queue full", even when capacity remained. The reported enterprise run (case 1392, result 27524) consumed 18,919,210 of 24,693,910 written rows before its consumer timed out. The source counts matched the writes. This PR isolates the materialized-tablet budget by queue and makes admission failures observable; it does not claim that this allocation change alone explains or resolves the entire observed delivery stall. ### Admission and recovery - Give each active consensus subscription queue an equal share of the DataNode materialization budget. A queue cannot reserve another queue's share. Existing allocations drain through delivery and ACK if a newly active queue reduces the share. - Preserve already accepted pending requests during ordinary backpressure. Cleanup and seek still clear stale pending requests, and rejected real-time requests remain eligible for WAL replay. - Return a critical poll error containing `SUBSCRIPTION_OVERSIZED_ENTRY` when one materialized WAL entry exceeds its queue's entire share. Do not advance that entry's WAL or commit progress. This is an intentional behavior change from the empty-budget oversized-entry exception; operators must increase the materialization budget, reduce concurrent queues, or narrow the topic to consume such an entry. ### Diagnostics and configuration - Report a stable `reasonCode` for real-time admission rejection, distinguishing subscription quota/limit, consensus request memory, writer backlog, queue capacity, and lifecycle states. Add queue-level memory used/quota and rejection counters to metrics and core reports. Capture offer rejection reasons per writer thread so concurrent offers cannot mislabel one another. - Explain in the config template and a Chinese user-guide page that `subscription_cache_memory_usage_percentage` controls serialized poll-response caching, not the materialized-tablet budget. Start at the default `0.2`; compare `0.3`/`0.4` only if response-cache eviction or reserialization is high and pipe memory has headroom. Size the ninth `chunk_timeseriesmeta_free_memory_proportion` share from the largest materialized entry, active queue count, and desired buffering depth. The repository has no published documentation navigation, so the new page still needs integration into the documentation site. ### Validation and limits - 56 focused tests passed: 54 DataNode and 2 consensus tests, including mixed-size queues, pending preservation, WAL recovery, oversized-entry errors, lifecycle cleanup, and concurrent rejection reporting. - Full English reactor `test` build and Chinese reactor `test-compile -DskipTests -P with-zh-locale` passed; Spotless and `git diff --check` passed. - The enterprise 3c1d/2-writer workload has not been rerun on this change. The 960 MB retained-tablet snapshot in result 27524 was taken after the consumer exited, so it does not by itself prove the cause of the earlier 67-second stop. Re-run that workload while collecting rejection codes, memory pools, prefetch-task durations, and ACK/GC timing. ### Related PR #18808 adds delivery-stall and node-memory diagnostics and retains the previous single oversized-entry exception. These PRs overlap in `ConsensusPrefetchingQueue`, metrics, memory manager, and localized messages. Their metric and oversized-entry semantics need reconciliation before merging both. <hr> This PR has: - [x] been self-reviewed for concurrent reads and writes. - [x] added documentation and Javadocs for the changed behavior. - [x] added focused unit tests for new paths. - [ ] been tested in a test IoTDB cluster with the reported enterprise workload. <hr> ##### Key changed classes `SubscriptionMemoryManager`, `ConsensusPrefetchingQueue`, `SubscriptionQueueRegistry`, and `ConsensusSubscriptionPrefetchingQueueMetrics`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
