JingsongLi commented on code in PR #9966:
URL: https://github.com/apache/paimon/pull/9966#discussion_r4185389909
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -281,6 +286,36 @@ static List<ManifestEntry> pruneByReadType(
if (anchor != null && !kept.contains(anchor)) {
kept.add(anchor);
}
+ // Blob and vector-store files may each cover only a sub-range of
their group. If the kept
+ // files are all dedicated files that do not cover the requested rows
(the whole group when
+ // there is no row-range pushdown), the reader would derive the range
from those sub-ranges
+ // and drop the rows outside them, so keep the full-range anchor.
+ if (anchor == null
+ && !kept.isEmpty()
+ && kept.stream()
+ .allMatch(
+ e ->
+ isBlobFile(e.file().fileName())
+ ||
isVectorStoreFile(e.file().fileName()))) {
+ ManifestEntry fullRangeAnchor = retrieveAnchorFile(group,
ManifestEntry::file);
Review Comment:
[P1] Preserve dedicated-only groups in incremental scans
A streaming DELTA group can legitimately contain only dedicated files: the
normal anchor remains in the previous snapshot and is absent from the APPEND
delta. With deletion vectors disabled, updating two BLOB columns over the same
row range produces two overlapping `.blob` files. This new unconditional
`retrieveAnchorFile` call then throws `IllegalStateException: A data-evolution
row range group must contain a normal ... anchor file`, stopping the stream
scan.
I reproduced this through `ReadBuilder.newStreamScan()`: write a normal-only
[0,1] range, consume the initial plan while projecting both BLOB columns,
commit both BLOB updates over [0,1], and plan the next delta. The baseline
returns the two update rows; this commit fails at this line. Please apply the
new anchor-retention logic only when the group contains a normal anchor,
preserve dedicated-only incremental groups, and add a DELTA regression test.
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionVectorReadPlanner.java:
##########
@@ -93,14 +93,28 @@ static List<ReadRange> plan(
}
}
- // Keep the sequential path for disjoint column groups, including
rolled vector files.
- if (!overlappingGroups) {
- return null;
- }
if (logicalRange == null) {
logicalRange = new Range(firstRowId, lastRowId);
}
+ // A projected vector store that covers only a sub-range of the group
cannot use the
+ // sequential path: its bunch has fewer rows than the anchor, so the
union reader rejects
+ // the mismatched row count. Plan explicit ranges so the uncovered
rows are NULL-filled.
+ boolean partialVector = false;
+ for (Candidate candidate : candidates) {
+ Range range = candidate.file.nonNullRowIdRange();
+ if (range.from > logicalRange.from || range.to < logicalRange.to) {
+ partialVector = true;
Review Comment:
[P2] Check coverage of the complete vector bunch before leaving the
sequential path
Comparing each physical vector file with the whole logical range marks
ordinary rolled files as partial even when their column bunch collectively
covers every row. The range-aware path then creates a separate union reader at
each vector-file boundary and reopens the same projected normal/blob bunches
for each range. The existing `testRolling` already has three vector files per
normal group.
On the same persisted 20,000-row table with an ID column, a 256-dimensional
FLOAT vector, and `vector.target-file-size=1 MB`, the writer produced one
normal file and 20 fully covering vector files. Compared with the baseline
planner, this commit increased Parquet normal-file opens from 1 to 20 and bytes
read from 80,269 to 1,605,620; returned rows and vector I/O were identical.
Please determine coverage across the complete projected column bunch so fully
covered rolled groups retain sequential reads, while genuinely uncovered ranges
still use NULL filling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
