zwy991114 commented on PR #66169:
URL: https://github.com/apache/doris/pull/66169#issuecomment-5327735846
## Upgrade/downgrade compatibility test
I tested the column-compression change across the following commits:
- PRE: 8305fe71867fb2acab5a769fe9a098ada59aff50
- The direct parent of the first per-column-compression commit.
- POST: 741b8b54c89d419cbcceba0b344086095f1ec9dd
- The latest commit containing this feature and the review fixes.
Both FE and BE runtime versions were verified before testing:
PRE FE/BE: 8305fe71867fb2acab5a769fe9a098ada59aff50
POST FE/BE: 741b8b54c89d419cbcceba0b344086095f1ec9dd
The binaries were built with ASAN. meta_tool from POST was used to inspect
all segment footers.
### Test setup
- One FE and one BE.
- PRE and POST shared the same:
- FE meta_dir
- BE storage_root_path
- replication_num = 1
- Automatic compaction was disabled to preserve the segment generated in
each phase.
- Each batch contained 4,096 rows with a 1,024-byte payload, so the data
exceeded the 256 KiB small-segment compression-suppression threshold.
- The column-compression table used:
- Table codec: LZ4F
- Column v: COMPRESSION ZSTD(9)
### Phase 1: PRE writes baseline data
Started PRE FE/BE, created an ordinary table using table-level ZSTD
compression, and wrote the first batch.
t_old_baseline batches=1
actual=4096,8386560,0,4095,4194304,4096,4096
expected=4096,8386560,0,4095,4194304,4096,4096
The tuple represents:
COUNT(*), SUM(k), MIN(k), MAX(k),
SUM(LENGTH(v)), COUNT(DISTINCT v), SUM(phase)
### Phase 2: upgrade to POST
POST successfully read the segment written by PRE and appended another
batch:
t_old_baseline batches=2
actual=8192,426373120,0,104095,8388608,8192,12288
expected=8192,426373120,0,104095,8388608,8192,12288
POST then created the per-column-compression table:
```
CREATE TABLE t_column_zstd9 (
k INT,
phase INT,
v VARCHAR(2048) COMPRESSION ZSTD(9)
)
DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"compression" = "LZ4F"
);
```
SHOW CREATE TABLE included the expected policy:
`v` varchar(2048) NULL COMPRESSION ZSTD(9)
"compression" = "LZ4F"
After writing the first batch:
t_column_zstd9 batches=1
actual=4096,8386560,0,4095,4194304,4096,4096
expected=4096,8386560,0,4095,4194304,4096,4096
The POST segment footer showed that only v used ZSTD level 9:
k: compression = LZ4F
phase: compression = LZ4F
v: compression = ZSTD
compression_level = 9
### Phase 3: downgrade to PRE
PRE successfully read both:
- The ordinary segments written by PRE and POST.
- The column segment written by POST using ZSTD level 9.
t_old_baseline batches=2
actual=8192,426373120,0,104095,8388608,8192,12288
expected=8192,426373120,0,104095,8388608,8192,12288
t_column_zstd9 batches=1
actual=4096,8386560,0,4095,4194304,4096,4096
expected=4096,8386560,0,4095,4194304,4096,4096
As expected, PRE did not render the unknown column policy:
`v` varchar(2048) NULL
"compression" = "LZ4F"
PRE then appended another batch to both tables:
t_old_baseline batches=3
actual=12288,1253959680,0,204095,12582912,12288,24576
expected=12288,1253959680,0,204095,12582912,12288,24576
t_column_zstd9 batches=2
actual=8192,426373120,0,104095,8388608,8192,12288
expected=8192,426373120,0,104095,8388608,8192,12288
The column table then contained two compatible segments:
POST-written segment:
v compression = ZSTD
compression_level = 9
PRE-written segment:
v compression = LZ4F
The PRE-written segment correctly fell back to the table codec because PRE
does not understand the per-column policy.
### Phase 4: re-upgrade to POST
POST successfully read every PRE/POST segment:
t_old_baseline batches=3
actual=12288,1253959680,0,204095,12582912,12288,24576
expected=12288,1253959680,0,204095,12582912,12288,24576
t_column_zstd9 batches=2
actual=8192,426373120,0,104095,8388608,8192,12288
expected=8192,426373120,0,104095,8388608,8192,12288
The column policy was visible again after re-upgrade:
`v` varchar(2048) NULL COMPRESSION ZSTD(9)
"compression" = "LZ4F"
POST appended the final batches. The final fingerprints were:
t_old_baseline batches=4
actual=16384,2491146240,0,304095,16777216,16384,40960
expected=16384,2491146240,0,304095,16777216,16384,40960
t_column_zstd9 batches=3
actual=12288,1253959680,0,204095,12582912,12288,24576
expected=12288,1253959680,0,204095,12582912,12288,24576
The final column-table footer sequence was:
Segment written by POST before downgrade:
v compression = ZSTD
compression_level = 9
Segment written by PRE during downgrade:
v compression = LZ4F
Segment written by POST after re-upgrade:
v compression = ZSTD
compression_level = 9
### Result
PASS
The test confirms:
1. POST can read and append to segments written by PRE.
2. PRE can read POST segments compressed with per-column ZSTD level 9.
3. Segments using different codecs can coexist in the same tablet.
4. During downgrade, PRE safely falls back to the table codec for new
writes.
5. After re-upgrade, POST restores the per-column policy and uses ZSTD
level 9 again.
6. All row counts and data fingerprints remain correct across PRE -> POST
-> PRE -> POST.
7. No ASAN error or residual FE/BE process was observed.
Scope note: this test used short version-switching phases and did not
intentionally force PRE FE to generate a metadata checkpoint while downgraded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
