eldenmoon opened a new pull request, #69221:
URL: https://github.com/apache/doris/pull/69221
### What problem does this PR solve?
Issue Number: CIR-22037 (related: apache/doris#66298)
VARIANT compaction can fail forever with `E-3113 string column length is too
large`.
`ColumnStr<uint32_t>` rejects more than 4 GiB of uncompressed chars, while
vertical compaction
plans output segments from **compressed** bytes. High-compression-ratio
VARIANT data reaches the
uint32 offset limit long before the planned segment reaches
`vertical_compaction_max_segment_size`.
The writer side made this deterministic: the VARIANT V2 writer buffered a
whole output segment in
memory before flushing it.
Reproduced on master with 1.2M rows x 4 KiB of VARIANT payload (4.9 GiB
uncompressed, ~8 MiB on
disk) and `vertical_compaction_max_segment_size=256MiB`:
```
[E-3113] string column length is too large: total_length=4295296512,
limit=4294967295, element_number=1044576, rows=1043584
VariantV2ColumnWriter::append -> ColumnStr<unsigned int>::insert_range_from
```
### What changed?
1. **Stream the root-only VARIANT writer**
(`variant/v2/variant_column_writer.*`).
The root column writer is created at `init()` and every input batch is
appended to it, so the
writer never retains a segment-sized `ColumnString` (`_root_jsonb` is
gone). `finalize()` only
records the row count; `estimate_buffer_size()` reports the current page
instead of the whole
segment.
2. **Bound the writers that still buffer** (`compaction/compaction.*`,
`common/config.*`).
When the compaction schema still selects a segment-buffering writer - the
shredder (no
extracted subcolumns, e.g.
`enable_vertical_compact_variant_subcolumns=false`) or
`VariantSubcolumnWriter` (extracted VARIANT subcolumns) - clamp the
per-segment row budget
using the input segments' VARIANT `raw_data_bytes` and
`variant_compaction_max_segment_uncompressed_bytes` (2 GiB default, 0
disables).
`raw_data_bytes` is the pre-encoding size the writer buffers;
`uncompressed_data_bytes` is the
encoded page size and can be orders of magnitude smaller for
dictionary-encoded strings
(measured 142,560 vs 4,944,000,000 for the same segment). The streaming
path is not clamped, so
its segment layout does not change.
Depends on #68444, which makes variant writers report
`raw/compressed/uncompressed` data page
bytes in the segment footer (first commit is that change, kept here so the
branch is testable).
### Tests
- `VariantColumnWriterReaderTest.test_variant_root_writer_streams_per_batch`
(new):
writes 4,000 x 4 KiB root scalars in 1,000-row batches and asserts the
buffered bytes stay below
two batches. Red/green verified locally with `doris_be_test`:
- red (buffering writer): `actual: 16428000 vs 8192000` -> FAILED
- green (streamed writer): `[ OK ] ... (156 ms)` -> PASSED
- Variant writer data page byte counters equal the bytes appended to the
file right after
`write_data()` (ordinary, extracted subcolumn, doc compact writer).
- `information_schema.column_data_sizes` reports VARIANT
compressed/uncompressed/raw bytes
(regression suite `test_query_sys_column_data_sizes_variant`).
- Manual end-to-end on a single-BE cluster (1.2M x 4 KiB):
- default path: **1 segment, no E-3113**, checksum unchanged
(count=1200000,
sum(len)=4927200000, sum(crc32)=2041942614000000), BE RSS peak +114 MiB
(was +5 GiB).
- `enable_vertical_compact_variant_subcolumns=false` (shredder): clamp
1,200,001 -> 521,233
rows per segment, **3 segments, no E-3113**, same checksum.
- **Sparse columns**: `variant_max_subcolumns_count=1` with a 4 KiB
`z_big` path that is not
materialized (sparse), extract off. Without the budget: E-3113 at
4,294,970,386 bytes
(`serialize_column_to_jsonb_vector`). With the budget: clamp 1,200,001
-> 517,465 rows per
segment, 3 segments, no E-3113, checksum identical to a fresh load
(count=1200000, sum(len)=4981200000, sum(crc32)=252636363600000).
### Remaining
- Non-vertical compaction does not use the segment row budget.
- Segments written before the footer raw-byte metadata existed (raw = 0)
cannot be estimated; the
streamed root path still covers them, the shredder path does not.
- Doc mode relies on the raw-byte budget.
### Release note
VARIANT compaction no longer buffers a whole output segment in one
`ColumnString`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
