eldenmoon opened a new pull request, #54508:
URL: https://github.com/apache/doris/pull/54508
This commit introduces a new mechanism to improve compaction
performance for tables with VARIANT columns by enabling vertical
compaction for frequently accessed subcolumns.
Key Changes:
1. Vertical Compaction of Subcolumns: During compaction, the table schema
is temporarily extended to treat "hot" subcolumns (frequently accessed paths)
within the VARIANT column as separate, top-level columns. This allows them to
be compacted more efficiently in their own column groups,
improving overall performance. This behavior is controlled by the new
configuration parameter
`enable_vertical_compact_variant_subcolumns`, which is enabled by default.
2. Schema Handling in Rowsets: * To prevent these temporary extracted
subcolumns from being persisted permanently, a new flag,
strip_variant_extracted_columns_in_rowset_meta, has been added to
RowsetWriterContext. When enabled, the RowsetWriter removes these temporary
columns from the schema before saving the rowset metadata. * This change
ensures that the in-memory Rowset object uses the same schema as the one
persisted in its RowsetMeta, preventing potential data inconsistencies.
3. Sparse Column Merging on Read:
* A new SparseColumnMergeIterator has been implemented. This iterator
is responsible for merging data from the base sparse representation of the
VARIANT with data from any extracted subcolumns during reads. This provides a
complete and unified view of the VARIANT data to the upper-level query engine.
4. Other Fixes and Improvements: * The DataTypeVariant constructor now
correctly accepts the max_subcolumns_count parameter. * A new correctness check
(check_path_stats) has been added to ensure that the final output rowset does
not contain any temporary, extracted subcolumns after compaction. * Previously
commented-out code related to subcolumn indexing and sparse column merging has
been re-enabled. * The build_basic_info method in the compaction process has
been refactored to return a Status for more robust error handling.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
