hoshinojyunn opened a new pull request, #66475:
URL: https://github.com/apache/doris/pull/66475
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Cloud tables previously had one table-level inverted_index_storage_format,
so an online V2-to-V3 migration required all partitions to move together. This
prevents a gradual rollout and cannot retain the format selected when an
existing partition was created while new partitions use a new default.
This PR adds the Cloud-only table property
partition.inverted_index_storage_format. It is the default for partitions
created after the property is set. The property supports V2 and V3; V1 is
rejected. Existing partitions and their tablet metadata are not rewritten by a
property update.
- Persist the resolved base-index inverted-index storage format in
PartitionInfo for every Cloud partition. The value is durable in journal replay
and is transferred through partition creation, recycle, recovery, and
replacement metadata.
- New explicit, automatic, and dynamic partitions use the current partition
format default. Existing partitions retain their recorded format, allowing V2
and V3 to coexist in one table.
- Tablet and rowset metadata persist the physical format independently of
shared schema metadata so that mixed-format partitions remain distinguishable
throughout a rolling upgrade.
- SHOW PARTITIONS, SHOW PARTITION, SHOW PARTITION ID, and the partitions TVF
expose InvertedIndexStorageFormat from PartitionInfo rather than reading tablet
metadata from MetaService for this value.
| Coverage area | Implementation coverage | Test coverage |
| --- | --- | --- |
| Property resolution and partition metadata |
partition.inverted_index_storage_format takes precedence, the legacy table
property remains the fallback, and the resolved format is persisted per Cloud
partition. | FE unit coverage for property analysis, table creation, table
properties, and partition persistence. |
| Partition creation paths | Explicit range/list partitions, auto
partitions, and dynamic partitions use the current resolved partition-format
default. | Cloud rollout regression coverage. |
| Mixed-format storage compatibility | Coexisting V2/V3 partitions retain
their own tablet and rowset metadata format. | BE rowset/tablet metadata unit
coverage and Cloud rowset-schema regression coverage. |
| Recycle, recovery, and replacement paths | The persisted partition format
survives recycle/recover, truncate, and insert-overwrite metadata paths. | FE
recycle/recover unit coverage and Cloud recycle/truncate regression coverage. |
| Metadata observability | Partition format is served from FE PartitionInfo,
avoiding per-partition tablet-meta reads for SHOW and TVF output. | Cloud
format regression coverage and partitions TVF coverage. |
### Release note
Cloud OLAP tables can roll out inverted-index file storage formats by
partition. New partitions can use V3 while existing V2 partitions remain online
and queryable.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
-
test_partition_cloud_inverted_index_storage_format_meta_write_switch
- [x] Unit Test
- FE partition-property, persistence, recycle/recover, and Cloud
schema-change tests
- BE rowset and tablet metadata tests
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. Cloud tables accept partition.inverted_index_storage_format;
it affects subsequently created partitions and exposes the resolved format in
partition metadata commands.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
