VGalaxies opened a new pull request, #19345:
URL: https://github.com/apache/iotdb/pull/19345
## Summary
Improves subscription delivery batching and tag-filter `IN` evaluation
performance on DataNode:
1. **Merge filtered-empty consensus entries into an active linger batch
(`ConsensusPrefetchingQueue`)**:
- Under low tag-filter selectivity (e.g., 1%–5%), most WAL/realtime
entries become empty (`tablets.isEmpty()`) after tag filtering. Previously,
every filtered-empty entry called `commitEmptyWriterEntry(...)`, which flushed
the active `lingerBatch` immediately and reduced average batch density to ~1
tablet per message.
- Filtered-empty entries are now appended to a non-empty `lingerBatch` as
zero-payload progress boundaries (preserving search index, physical time, and
local seq advancement upon batch ack) while still flushing immediately when the
batch has already reached `maxTablets` or `maxBatchBytes`.
- `DeliveryBatchState` tracks `volatile replayEntryCount` (non-empty
entries only) for `getRemainingEventCount()`, `getLag()`, and
`DeliveryBatchRecord`, and collapses consecutive trailing filtered-empty
entries in place to keep memory bounded during long empty runs.
2. **Pre-compile `IN` literal lists into immutable `HashSet`s and make
`MAX_IN_VALUES = 256` reachable (`TagFilterValidator` / `TagFilterMatcher` /
`TagFilterEvaluator`)**:
- `TagFilterValidator#validateAndCompile` pre-builds immutable
`Set<String>` maps for `InPredicate` nodes so
`TagFilterEvaluator#visitInPredicate` performs `O(1)` lookup instead of linear
scans over the literal list on every row/tablet.
- `InListExpression` literal elements are no longer double-charged
against `MAX_AST_NODES = 256` (they are already bounded by `MAX_IN_VALUES =
256`), making 256-value `IN` lists usable instead of failing at ~254 elements.
## Concrete Before-and-After Examples
- **Filtered-empty batch merging** (1%–5% tag-filter hit rate, 1 non-empty
entry followed by 20 filtered-empty entries):
- Before: the first filtered-empty entry invokes
`commitEmptyWriterEntry(...)`, flushing the 1-tablet `lingerBatch` immediately;
average batch size collapses to ~1 tablet/message.
- After: filtered-empty entries ride along in `lingerBatch` without
triggering an early flush (unless `maxTablets` or `maxBatchBytes` is already
reached), keeping batch density 3–4x higher while advancing committed progress
accurately on ack and reporting non-inflated lag.
- **`TagFilter` 256-value `IN` predicate** (`device_id IN ("d0", ...,
"d255")`):
- Before: rejected during validation because the 256 `StringLiteral` nodes
plus the `InPredicate`/`InListExpression` exceeded `MAX_AST_NODES = 256`;
smaller lists scanned linearly (`O(K)` per check).
- After: accepted up to 256 values (257 still rejected by `MAX_IN_VALUES =
256`) and evaluated in `O(1)` via pre-compiled `HashSet`.
## Verification
- `mvn test -pl iotdb-core/datanode -am
-Dsurefire.failIfNoSpecifiedTests=false
-Dtest=TagFilterParserTest,TabletTagFilterTest,ConsensusLogToTabletConverterTest,ConsensusPrefetchingQueueWriterOrderingTest,ConsensusPrefetchingQueueDeliveryTest,ConsensusPrefetchingQueueTest`:
- `ConsensusPrefetchingQueueWriterOrderingTest`: 15/15 passed
- `ConsensusLogToTabletConverterTest`: 22/22 passed
- `TabletTagFilterTest`: 8/8 passed
- `TagFilterParserTest`: 7/7 passed
- `ConsensusPrefetchingQueueTest`: 57/57 passed
- `ConsensusPrefetchingQueueDeliveryTest`: 15/16 passed (1 pre-existing
reflection `NoSuchMethod` on `PendingSeekRequest.<init>` in
`testSeekResetDiscardsDetachedStateAndRejectsOldCallbacksAndAcks`, unrelated to
this PR)
- `mvn test-compile -pl iotdb-core/datanode -am -DskipTests`: exit 0
- `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale
-DskipTests`: exit 0
- `mvn spotless:check -pl iotdb-core/datanode`: exit 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]