VGalaxies opened a new pull request, #17936:
URL: https://github.com/apache/iotdb/pull/17936
## Purpose
Add a `column-filter` topic attribute for table-model subscriptions so a
topic can
restrict which columns consumers receive, based on column metadata, applied
uniformly
across historical, real-time pipe, and IoTConsensus WAL paths.
## What
- New optional topic attribute `column-filter` (table-model only); unset ==
`true`.
Restricted boolean predicate over column-metadata fields aligned with
`information_schema.columns`: `database`, `table_name`, `column_name`,
`datatype`,
`category`. Supports `=`/`!=`, `IN`, `LIKE`, `REGEXP`, `IS [NOT] NULL`,
`AND`/`OR`/`NOT`.
- Independent `ColumnFilter.g4` grammar (decoupled from the query grammar)
producing the
reused relational AST nodes; a whitelist validator rejects unsupported
syntax/fields.
- Resolved to a per-(database, table) column set at CREATE/ALTER (and on
re-bind) against the
current schema, cached per topic on the DataNode and used for O(1) runtime
pruning.
TAG columns are retained; ATTRIBUTE columns are not transmittable.
- Applied in all paths: IoTConsensus WAL converter, pipe tablet batch, and
TsFile reseal
(prune at tablet granularity; raw TsFile passthrough kept only when the
filter is trivial).
- `timeSelected` propagated as an optional, backward-compatible per-table
flag on the poll
response (defaults to exposing time for old consumers).
- ALTER TOPIC hot-refreshes the cached matcher (no converter rebuild;
in-flight not reprocessed).
- Removed the legacy consensus-only `column` regex attribute.
## Tests
- Unit: parser/validator/evaluator, binder, tablet pruner, topic config,
poll response, record handler.
- Integration (table model): live/snapshot, record & TsFile,
multi-consumer-group, historical+
realtime consistency, ALTER rebind, empty-result progress, snapshot
semantics, custom time
column, altered data type, tree-view source-column binding, attribute
drop, 1C3D cluster
ALTER-after-owner-transfer, DataNode restart re-snapshot,
invalid-expression rejection, permission.
## Notes
- The column-filter bound set is a non-persisted snapshot: stable between
binds, re-derived
from the current schema on CREATE/ALTER/restart/owner-transfer; not frozen
for the topic lifetime.
- Tree-model topics are unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
