eldenmoon opened a new pull request, #67868:
URL: https://github.com/apache/doris/pull/67868
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Vertical compaction extends a VARIANT column into extracted subcolumns. When
a selected path
cannot get a concrete type from segment metadata (it was stored in the
sparse column of some
input segment, has no subcolumn type stats, or the sparse path stats are
truncated), the
extracted column is VARIANT-typed and is written by
`VariantSubcolumnWriter`. That writer
buffers every row of the output segment in memory so it can infer the least
common type in
`finalize()`, and only then creates the real column writer.
The output segment row count is estimated from the compressed input size, so
a highly
compressible JSON string path can put more than 4 GiB of string data into
one segment. The
buffered `ColumnString` then hits its uint32 offset limit and compaction
fails with
`[E-3113] string column length is too large`, retrying the same version
range forever.
Reproduced on master with 200,000 rows whose `big` path carries ~24 KB
strings (4.74 GB in
total) and is sparse in the early rowsets: full compaction failed with
`total_length=4300785738, limit=4294967295` inside
`VariantSubcolumnWriter::_append`. When the
same data gets a concrete STRING type, `ScalarColumnWriter` streams it and
compaction succeeds.
This PR makes the V2 input path of `VariantSubcolumnWriter` stream once its
buffer grows past
the new mutable BE config `variant_subcolumn_stream_write_threshold_bytes`
(default 2 GiB):
- the storage type is locked at that point: a predefined path keeps its
storage type, any other
path uses nullable JSONB, which can hold every later value without
rewriting written pages;
- the buffered rows are written through the real column writer, and every
later batch is
converted and written immediately, so the buffer never holds more than one
batch;
- `finalize()` writes the remaining rows and records `num_rows` /
`none_null_size`.
Segments whose buffered data stays below the threshold keep the existing
precise type
inference, and the type preparation / writer creation code is shared by both
paths. The legacy
V1 input (`ColumnVariant`) path is unchanged because it is going to be
removed.
### Release note
Add BE config `variant_subcolumn_stream_write_threshold_bytes`. A
VARIANT-typed extracted
subcolumn whose buffered V2 data in one segment exceeds it is written in a
streaming way and
stored as JSONB (or its predefined type) in that segment.
### Check List (For Author)
- Test: Unit Test
- Added
`VariantColumnWriterReaderTest.v2_extracted_subcolumn_writer_streams_after_buffer_threshold`
and
`VariantColumnWriterReaderTest.v2_typed_extracted_subcolumn_writer_streams_with_storage_type`.
- Red: without the writer change both tests failed only on the type-lock
assertion (the
column type after each batch stayed VARIANT); the data read-back
assertions passed.
- Green: with the change both tests pass. Filter
`*VariantColumnWriterReaderTest*:*VariantPathBuilderTest*:*VariantShredderTest*:*VariantWriterCompatibilityTest*:*VariantSpecializedWriterCompatibilityTest*:*SchemaUtilRowsetTest*`
ran 109 tests: 107 passed, 2 skipped (NestedGroup write path is not
available in this build).
- Manual test: the E-3113 reproduction above was done on master; on
current master that
compaction still feeds legacy `ColumnVariant` input to this writer, so
it is not changed by
this PR.
- Behavior changed: Yes. A path that exceeds the threshold within one
segment is stored with the
locked type (JSONB unless predefined) in that segment instead of failing
compaction.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01AU69xp5L6BqZHq7FKzJ9Ts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
