LuciferYang commented on code in PR #9966:
URL: https://github.com/apache/paimon/pull/9966#discussion_r4195867399
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -281,6 +286,36 @@ static List<ManifestEntry> pruneByReadType(
if (anchor != null && !kept.contains(anchor)) {
kept.add(anchor);
}
+ // Blob and vector-store files may each cover only a sub-range of
their group. If the kept
+ // files are all dedicated files that do not cover the requested rows
(the whole group when
+ // there is no row-range pushdown), the reader would derive the range
from those sub-ranges
+ // and drop the rows outside them, so keep the full-range anchor.
+ if (anchor == null
+ && !kept.isEmpty()
+ && kept.stream()
+ .allMatch(
+ e ->
+ isBlobFile(e.file().fileName())
+ ||
isVectorStoreFile(e.file().fileName()))) {
+ ManifestEntry fullRangeAnchor = retrieveAnchorFile(group,
ManifestEntry::file);
Review Comment:
Fixed in adbd0b2: the anchor is only added when the group has a normal file,
so dedicated-only DELTA groups are returned as is. Your repro is now
`MultipleBlobTableTest#testStreamScanReadsBlobOnlyDelta`.
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionVectorReadPlanner.java:
##########
@@ -93,14 +93,28 @@ static List<ReadRange> plan(
}
}
- // Keep the sequential path for disjoint column groups, including
rolled vector files.
- if (!overlappingGroups) {
- return null;
- }
if (logicalRange == null) {
logicalRange = new Range(firstRowId, lastRowId);
}
+ // A projected vector store that covers only a sub-range of the group
cannot use the
+ // sequential path: its bunch has fewer rows than the anchor, so the
union reader rejects
+ // the mismatched row count. Plan explicit ranges so the uncovered
rows are NULL-filled.
+ boolean partialVector = false;
+ for (Candidate candidate : candidates) {
+ Range range = candidate.file.nonNullRowIdRange();
+ if (range.from > logicalRange.from || range.to < logicalRange.to) {
+ partialVector = true;
Review Comment:
Fixed in adbd0b2. The planner now checks each projected column's whole
bunch: it replays `VectorFileBunch`'s file selection and stays sequential when
the kept files tile the group. Rolled files stay sequential, also when
row-range pushdown prunes some of them.
A union of the ranges was not enough: a newer update over part of an older
file also spans the group, but the bunch keeps only the newer file
(`VectorStoreTableTest#testNewerVectorUpdateOverPartOfTheColumn`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
