eldenmoon opened a new pull request, #70396:
URL: https://github.com/apache/doris/pull/70396
### What problem does this PR solve?
Issue Number: None
Related PR: #67714, #70188 (earlier attempts; this PR replaces both with a
smaller design)
Problem Summary:
Users cannot append schema template entries or change
`variant_max_subcolumns_count` on an existing VARIANT
column, or add/drop a `field_pattern` inverted index, without a heavy schema
change. This PR makes all of them
metadata-only changes.
Design: templates and the subcolumn budget are a write policy.
- FE: `VariantPropertyChangePolicy` allows a budget change and appending
template entries on ordinary VARIANT
columns (DOC mode, NestedGroup and typed_paths_to_sparse columns keep
rejecting them; template changes on
row-store tables are rejected). Existing entries must stay an identical
prefix; every entry-level decision goes
through one hook so a later narrow-to-wide type change only relaxes that
hook. A property change cannot be
combined with nullability, default, comment, key, aggregation or position
changes. `VariantTemplateIndexBinder`
is the single place that binds `field_pattern` indexes to templates for
CREATE TABLE, ALTER and query index
selection; ADD/DROP INDEX with `field_pattern` now use the ordinary light
index change path. `Column` now copies
and compares `fieldPatternType`.
- BE compaction: when an input rowset's Variant templates or budget differ
from the compaction target, that
compaction writes Variant columns in full-root mode and re-shreds old rows
exactly as a new load would (old
values are cast to the template type; a value that cannot be cast becomes
NULL, as in a load). The path count
check is skipped for that compaction. Once the output carries the target
schema, later compactions use the
normal extended-schema path again.
- BE read: for a Variant subcolumn whose stored type in a segment differs
from the read type, storage
predicates, zone maps, bloom filters and indexes are not applied; the
conjunct, which the scan keeps for
non-key columns, filters the cast values. A field_pattern index added
after a segment was written follows the
existing missing-index fallback
(`enable_fallback_on_missing_inverted_index`).
Behaviour to note: before such a compaction, old rows of an appended
template path read back as stored (for
example `"001"`); after it, they read back as a load after the ALTER would
(`1`). New rows never change.
### Release note
Support appending VARIANT schema template entries, changing
`variant_max_subcolumns_count`, and adding or dropping
`field_pattern` inverted indexes as lightweight ALTER operations. Existing
data is converted to the new templates
lazily by compaction.
### Check List (For Author)
- Test
- [x] Regression test: `variant_p0/light_alter` (template append, budget
changes, field_pattern ADD/DROP INDEX,
rejections; each compaction result is compared with a control table
loaded after the ALTER),
`variant_p0/predefine` (28 suites), `variant_p0/schema_change` (9
suites), `test_predefine_ddl` updated for
the new behaviour.
- [x] Unit Test: FE `VariantPropertyChangePolicyTest`, `ColumnTest`,
`SchemaChangeHandlerTest`,
`CreateTableCommandTest`, `OlapTableTest`; BE
`IndexStorageVariantCompactionSchemaTest`,
`VariantAlterCompactionTest` (both vertical modes),
`SegmentPredicateTemplateAlterTest`,
`ScanNormalizePredicate`, and the existing SchemaUtil /
VariantCompaction / VariantColumnReader /
SegmentIterator suites (247 passed).
- Cloud: same code path, not run on a Cloud cluster.
- Behavior changed: Yes (new ALTER capabilities)
- Does this need documentation: Yes
Known limitation: DROP INDEX of a `field_pattern` index removes it from the
schema immediately; per-path index
data in existing segments is reclaimed when those segments are rewritten by
compaction.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
