Caideyipi opened a new pull request, #18925:
URL: https://github.com/apache/iotdb/pull/18925

   ## Description
   
   Table subscription column filters currently bind to a schema snapshot when 
the topic matcher is initialized. A filter such as `column_name = "s2"` retains 
the original TAGs but omits TAGs added while the consumer is running. New 
matching FIELDs and tables also require rebinding.
   
   Make column filters follow schema changes by default. Selecting part of a 
table's FIELDs automatically retains all its TAGs, including later additions, 
without ALTER TOPIC or a consumer restart. Matching FIELDs and tables are 
discovered dynamically, and datatype predicates use the current committed table 
schema. Preserve view aliases, ATTRIBUTE exclusion, per-table TIME selection, 
and filters that match no columns.
   
   The expression is parsed once. Each Tablet resolves a cached table plan 
through the local committed schema cache; the plan is rebuilt only when that 
table's schema instance changes. Retain runtime TAGs in the existing pruning 
loop and replace per-column table lookups with direct set membership. The local 
schema accessor preserves metadata lease fencing and does not contact 
ConfigNode.
   
   Document the behavior in `iotdb-client/subscription/README.md`. There is no 
fixed column-schema snapshot option. The topic's `mode = snapshot` continues to 
control the historical data cutoff independently of column filtering.
   
   ## Validation
   
   - 43 unit tests passed: `ColumnFilterBinderTest`, `ColumnFilterParserTest`, 
`TabletColumnPrunerTest`, `SubscriptionTopicAgentTest`, 
`DataNodeTableCacheTest`, and `DataNodeTableCacheLeaseTest`.
   - Five changed/new methods in `IoTDBSubscriptionColumnFilterIT` passed: new 
tables, new FIELDs, new TAGs with selected FIELDs, new TAG values in TsFile 
payloads, and live datatype changes. Each case warms the matcher before the 
schema change.
   - 
`IoTDBConsensusSubscriptionColumnFilterClusterIT#testStreamingColumnFilterIncludesNewTagsAndFieldsOnThreeDataNodes`
 passed with 1 ConfigNode, 3 DataNodes, and data replication factor 2.
   - DataNode and integration-test compilation, Checkstyle, Spotless, and `git 
diff --check` passed.
   
   Integration-test condition: master currently compiles 
`CommonConfig.SUBSCRIPTION_ENABLED` as `false`, so stock test artifacts reject 
topic creation. The six integration tests ran with that flag enabled only in 
temporary generated test artifacts, recompiling `DataNodeMemoryConfig` to 
account for the inlined constant. The original generated jars were restored 
afterward; the production feature gate is unchanged by this PR.
   
   ## Performance
   
   A local pruning microbenchmark compared the original snapshot matcher/pruner 
from the base commit with the streaming implementation and a real 
`DataNodeTableCache`. After warmup, nine alternating samples were measured per 
case; the table reports medians in ns/Tablet.
   
   | Columns | Rows/Tablet | Snapshot | Streaming |
   | --- | --- | --- | --- |
   | 8 | 1 | 1119.8 | 1159.1 |
   | 8 | 1024 | 586.5 | 399.7 |
   | 32 | 1 | 1760.7 | 1073.5 |
   | 32 | 1024 | 1727.3 | 1057.8 |
   | 128 | 1 | 7134.6 | 4037.6 |
   | 128 | 1024 | 7335.9 | 4231.9 |
   
   The narrow single-row case adds about 39 ns (3.5%); the other measured cases 
spend less time pruning. These results cover pruning with unchanged schema, not 
end-to-end throughput. Given the small local lookup cost and reuse of table 
plans, no legacy schema-snapshot configuration is retained.
   
   This PR has:
   
   - [x] been self-reviewed, including concurrent schema reads and table-plan 
replacement.
   - [x] added documentation for modified behavior.
   - [x] added comments and Javadocs for the table-plan cache and local schema 
accessor.
   - [x] added or updated unit tests and integration tests.
   - [x] been tested in IoTDB test clusters under the integration-test 
condition above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to