hoshinojyunn opened a new pull request, #66235:
URL: https://github.com/apache/doris/pull/66235
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Cloud tables previously used one table-level
`inverted_index_storage_format`, so an online V2-to-V3 migration required all
partitions to move together. This prevents a gradual rollout and makes it
impossible to retain the correct tablet schema for old partitions while
creating new partitions with a new format.
This PR adds the Cloud-only table property
`partition.inverted_index_storage_format`. It is the format default for
partitions created after the property is set. The property supports V2 and V3;
V1 is rejected. Existing partitions and their tablet metas are not rewritten.
Design:
- Persist the resolved base-index inverted-index storage format in
`PartitionInfo` for every Cloud partition. The value is durable in
journal/replay and is transferred through partition replacement,
recycle/recover, add-partition-like, insert overwrite, and truncate flows.
- New explicit, automatic, and dynamic partitions use the current partition
format default. Existing partitions continue to use their recorded format,
allowing V2 and V3 to coexist in one table.
- A replacement partition reads the source partition's actual tablet metas
and preserves each source index schema, schema hash, and schema version. This
prevents truncate and insert overwrite from accidentally applying a newer table
schema or format to an old partition.
- A Cloud schema change collects the distinct active base-index formats. It
allocates one base shadow schema version per format, persists the
format-to-version map in `CloudSchemaChangeJobV2`, and creates each partition's
base shadow tablet with the matching format and schema version. Non-base
indexes keep their normal schema-change behavior.
- `SHOW PARTITIONS`, `SHOW PARTITION <id>`, `SHOW PARTITION ID`, and the
partitions TVF expose `InvertedIndexStorageFormat` from `PartitionInfo`; they
no longer need to read tablet meta from MetaService for this value.
Example SQL and expected effect:
```sql
CREATE TABLE inv_format_rollout (
k DATE NOT NULL,
v VARCHAR(100),
INDEX idx_v (v) USING INVERTED
)
DUPLICATE KEY(k)
PARTITION BY RANGE(k) (
PARTITION p_old VALUES LESS THAN ("2024-01-01")
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"inverted_index_storage_format" = "V2"
);
ALTER TABLE inv_format_rollout
SET ("partition.inverted_index_storage_format" = "V3");
ALTER TABLE inv_format_rollout
ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01"));
SHOW PARTITIONS FROM inv_format_rollout;
```
Selected output:
```
+---------------+----------------------------+
| PartitionName | InvertedIndexStorageFormat |
+---------------+----------------------------+
| p_old | V2 |
| p_new | V3 |
+---------------+----------------------------+
```
Both formats remain queryable during the rollout:
```sql
INSERT INTO inv_format_rollout VALUES
("2023-01-01", "old token"),
("2024-01-01", "new token");
SELECT CAST(k AS STRING), v
FROM inv_format_rollout
WHERE v MATCH_ANY "token"
ORDER BY k;
```
The query returns both rows. `TRUNCATE TABLE ... PARTITION p_old` and
`INSERT OVERWRITE ... PARTITION(p_old)` continue to create V2 tablet metas for
`p_old`, even after the default changes to V3. A subsequent `MODIFY COLUMN`
creates distinct base shadow schema versions for active V2 and V3 partitions,
then preserves the format/version pairing when committing the schema change.
### Release note
Cloud OLAP tables can roll out inverted-index file storage formats by
partition. New partitions can use V3 while existing V2 partitions remain online
and queryable.
### Check List (For Author)
- Test
- [x] Regression test
- `test_cloud_show_inverted_index_storage_format`:
range/list/auto/dynamic partition format display through SHOW and tablet meta.
-
`test_partition_cloud_inverted_index_storage_format_recycle_truncate`:
drop/recover and truncate preserve V2/V3 partition formats and inverted-index
queryability.
- `test_partition_cloud_inverted_index_storage_format_rollout`:
V2-to-V3 rollout, add partition, auto/dynamic partition creation, truncate,
insert overwrite, schema change, rollup, and recovery with mixed formats.
- [x] Unit Test
- FE unit coverage for property validation, table creation,
partition persistence/replay, replacement, recycle/recover, insert overwrite,
Cloud tablet-meta creation, schema-change version allocation, and SHOW output.
- Ran the changed FE test classes (70 tests) and targeted
`OlapTableTest` plus `CatalogRecycleBinTest` (41 tests), all passing.
- Behavior changed:
- [x] Yes. Cloud tables accept
`partition.inverted_index_storage_format`; it affects only subsequently created
partitions and exposes the resolved format in partition metadata commands.
- Does this need documentation?
- [x] No. The SQL contract and rollout behavior are documented in this
PR description.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
