VGalaxies opened a new pull request, #19345:
URL: https://github.com/apache/iotdb/pull/19345

   ## Summary
   Improves subscription delivery batching and tag-filter `IN` evaluation 
performance on DataNode:
   
   1. **Merge filtered-empty consensus entries into an active linger batch 
(`ConsensusPrefetchingQueue`)**:
      - Under low tag-filter selectivity (e.g., 1%–5%), most WAL/realtime 
entries become empty (`tablets.isEmpty()`) after tag filtering. Previously, 
every filtered-empty entry called `commitEmptyWriterEntry(...)`, which flushed 
the active `lingerBatch` immediately and reduced average batch density to ~1 
tablet per message.
      - Filtered-empty entries are now appended to a non-empty `lingerBatch` as 
zero-payload progress boundaries (preserving search index, physical time, and 
local seq advancement upon batch ack) while still flushing immediately when the 
batch has already reached `maxTablets` or `maxBatchBytes`.
      - `DeliveryBatchState` tracks `volatile replayEntryCount` (non-empty 
entries only) for `getRemainingEventCount()`, `getLag()`, and 
`DeliveryBatchRecord`, and collapses consecutive trailing filtered-empty 
entries in place to keep memory bounded during long empty runs.
   2. **Pre-compile `IN` literal lists into immutable `HashSet`s and make 
`MAX_IN_VALUES = 256` reachable (`TagFilterValidator` / `TagFilterMatcher` / 
`TagFilterEvaluator`)**:
      - `TagFilterValidator#validateAndCompile` pre-builds immutable 
`Set<String>` maps for `InPredicate` nodes so 
`TagFilterEvaluator#visitInPredicate` performs `O(1)` lookup instead of linear 
scans over the literal list on every row/tablet.
      - `InListExpression` literal elements are no longer double-charged 
against `MAX_AST_NODES = 256` (they are already bounded by `MAX_IN_VALUES = 
256`), making 256-value `IN` lists usable instead of failing at ~254 elements.
   
   ## Concrete Before-and-After Examples
   - **Filtered-empty batch merging** (1%–5% tag-filter hit rate, 1 non-empty 
entry followed by 20 filtered-empty entries):
     - Before: the first filtered-empty entry invokes 
`commitEmptyWriterEntry(...)`, flushing the 1-tablet `lingerBatch` immediately; 
average batch size collapses to ~1 tablet/message.
     - After: filtered-empty entries ride along in `lingerBatch` without 
triggering an early flush (unless `maxTablets` or `maxBatchBytes` is already 
reached), keeping batch density 3–4x higher while advancing committed progress 
accurately on ack and reporting non-inflated lag.
   - **`TagFilter` 256-value `IN` predicate** (`device_id IN ("d0", ..., 
"d255")`):
     - Before: rejected during validation because the 256 `StringLiteral` nodes 
plus the `InPredicate`/`InListExpression` exceeded `MAX_AST_NODES = 256`; 
smaller lists scanned linearly (`O(K)` per check).
     - After: accepted up to 256 values (257 still rejected by `MAX_IN_VALUES = 
256`) and evaluated in `O(1)` via pre-compiled `HashSet`.
   
   ## Verification
   - `mvn test -pl iotdb-core/datanode -am 
-Dsurefire.failIfNoSpecifiedTests=false 
-Dtest=TagFilterParserTest,TabletTagFilterTest,ConsensusLogToTabletConverterTest,ConsensusPrefetchingQueueWriterOrderingTest,ConsensusPrefetchingQueueDeliveryTest,ConsensusPrefetchingQueueTest`:
     - `ConsensusPrefetchingQueueWriterOrderingTest`: 15/15 passed
     - `ConsensusLogToTabletConverterTest`: 22/22 passed
     - `TabletTagFilterTest`: 8/8 passed
     - `TagFilterParserTest`: 7/7 passed
     - `ConsensusPrefetchingQueueTest`: 57/57 passed
     - `ConsensusPrefetchingQueueDeliveryTest`: 15/16 passed (1 pre-existing 
reflection `NoSuchMethod` on `PendingSeekRequest.<init>` in 
`testSeekResetDiscardsDetachedStateAndRejectsOldCallbacksAndAcks`, unrelated to 
this PR)
   - `mvn test-compile -pl iotdb-core/datanode -am -DskipTests`: exit 0
   - `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale 
-DskipTests`: exit 0
   - `mvn spotless:check -pl iotdb-core/datanode`: exit 0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to