hoshinojyunn opened a new pull request, #66017:
URL: https://github.com/apache/doris/pull/66017
### What problem does this PR solve?
Issue Number: None
Related PR: #57013
Problem Summary:
An inverted-index storage-format rollout must not rewrite historical
tablets. A table can therefore contain V2 partitions created before the rollout
and V3 partitions created afterwards. Before this change, the format selected
for a future partition was not persisted as partition metadata, so lifecycle
paths could not consistently preserve the format that the partition requires.
This PR introduces `partition.inverted_index_storage_format` for future
logical partitions. It resolves and persists V2 or V3 in `PartitionInfo`, and
passes that value to tablet and replica creation. Changing the property does
not modify existing partitions.
Implementation changes:
- Add the partition-scoped property; V1 is rejected and V2/V3 are supported.
- Preserve the resolved format for initial, manual, auto, and dynamic
partitions.
- Carry it through edit-log replay, restore, recycle-bin recovery, truncate,
insert overwrite, and replica recovery.
- Expose the partition format in `SHOW PARTITIONS`, `SHOW PARTITION
<partition_id>`, `partitions(...)`, and `SHOW CREATE TABLE`.
Example SQL:
```sql
CREATE TABLE db.t (
k DATE NOT NULL,
v VARCHAR(100),
INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english")
)
ENGINE=OLAP
DUPLICATE KEY(k)
PARTITION BY RANGE(k) (
PARTITION p_old VALUES LESS THAN ("2024-01-01")
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"inverted_index_storage_format" = "V2"
);
ALTER TABLE db.t
SET ("partition.inverted_index_storage_format" = "V3");
ALTER TABLE db.t
ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01"));
SHOW PARTITIONS FROM db.t;
SHOW TABLETS FROM db.t PARTITION(p_new);
```
Observability result:
| Partition | FE metadata (`SHOW PARTITIONS` / `partitions(...)`) | BE
tablet header (`GET <MetaUrl>`) |
| --- | --- | --- |
| `p_old` | `InvertedIndexStorageFormat = V2` |
`schema.inverted_index_storage_format = "V2"` |
| `p_new` | `InvertedIndexStorageFormat = V3` |
`schema.inverted_index_storage_format = "V3"` |
`SHOW TABLETS FROM db.t PARTITION(p_new)` provides each tablet MetaUrl.
Requesting the URL verifies the format stored in the actual BE tablet metadata,
not only the FE partition metadata.
### Release note
Users can roll out V3 inverted-index storage format to newly created
partitions without changing historical partitions.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Test summary:
- Passed `./run-fe-ut.sh --run
org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest`.
- Passed `DISABLE_BUILD_UI=ON ./build.sh --fe`, including FE Checkstyle and
packaging.
- Passed `./run-regression-test.sh --run -d inverted_index_p0 -s
test_partition_inverted_index_storage_format_rollout`. It covers initial,
manual, auto, and dynamic partitions; FE observability; and actual BE tablet
meta headers.
- Passed `./run-regression-test.sh --run -d inverted_index_p2 -s
test_partition_inverted_index_storage_format_replica_recovery`. It verifies
that a recovered V2 replica remains V2 while a V3 partition remains V3.
- Cloud tests were not run.
- Behavior changed:
- [ ] No.
- [x] Yes. Newly created partitions can use a different inverted-index
storage format from historical partitions.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
