leaves12138 opened a new pull request, #9092:
URL: https://github.com/apache/paimon/pull/9092
### Purpose
Reduce memory usage when detecting sorted global indexes that need to be
refreshed after data-evolution column updates.
Previously, `SortedGlobalIndexScanner` materialized all data manifest
entries, and the refresh planner retained matching `DataFileMeta` objects
before sorting them. On tables with many updated files, refresh planning memory
therefore grew with the number of manifest entries and matching files.
This change:
- scans data manifests through reusable `BinaryManifestEntry` projections;
- tracks removed data files with `DeletedIdentifierSet`;
- merges updated row-id ranges by distinct index scan watermarks instead of
retaining file metadata;
- keeps the existing entry-based planner API through a streaming iterator
path; and
- adds coverage for column updates, binary/entry scan parity, and DELETE
entries produced by compaction.
The user-visible behavior is unchanged, while incremental global-index
refresh planning uses substantially less memory.
### Tests
- `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none
-Dtest=SortedGlobalIndexScannerTest test`
- 11 tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
