Gabriel39 opened a new pull request, #65451:
URL: https://github.com/apache/doris/pull/65451
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Paimon and Iceberg deletion vectors were expanded into `vector<int64_t>`
before entering the query-local decoded cache. That representation wastes
memory for sparse DVs and makes cache reuse more expensive than retaining the
source bitmap.
This PR changes both legacy readers and format v2 to:
- keep decoded Paimon/Iceberg DVs as `Roaring64Map` in the query-local cache;
- keep remote Puffin/delete-file reads on Doris's normal
`CachedRemoteFileReader` path so their bytes can be persisted in the disk File
Cache;
- consume Roaring directly in ORC and format v2, while legacy Parquet only
materializes deleted positions for the current row group;
- report decoded-DV cache hit/miss counters separately from the underlying
disk File Cache local/remote/peer I/O counters;
- preserve the existing vector representation for position delete files,
including the case where an Iceberg split has both position deletes and a DV.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
Added/updated UT coverage for:
- Paimon DV decoding retaining Roaring representation in V1 and format v2;
- decoded cache miss-to-hit reporting without rerunning the file/decode
creator;
- format-v2 `DeletePredicate` filtering directly from a compressed Roaring
DV;
- existing malformed Paimon/Iceberg DV error paths.
Validation performed:
- `build-support/check-format.sh`
- clang-format 16 `--dry-run --Werror` on all changed C/C++ files
- `git diff --check`
Per request, no local compilation or dependency installation was performed.
Any temporary dependency download was stopped and removed before this PR was
created.
- Behavior changed:
- [ ] No.
- [x] Yes. Decoded deletion vectors retain their compressed Roaring
representation and cache-layer metrics are separated.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
