LuciferYang commented on code in PR #9966:
URL: https://github.com/apache/paimon/pull/9966#discussion_r4105624934
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -290,6 +318,35 @@ static List<ManifestEntry> pruneByReadType(
: kept;
}
+ /**
+ * Whether the row-id ranges of {@code entries} together cover every
{@code target} range
+ * (inclusive), with no gap. Dedicated files are always within the group's
full range, so
+ * covering the requested targets means the reader can derive the range
from them without the
+ * anchor.
+ */
+ private static boolean coversRanges(List<ManifestEntry> entries,
List<Range> targets) {
Review Comment:
Fixed. Coverage is now judged per projected column instead of by unioning
files across columns, so a projection whose columns cover disjoint sub-ranges
(b1 over [0,6], b2 over [3,9]) keeps the anchor. Added
`DataEvolutionFileStoreScanTest#testReadTypePruningKeepsAnchorForMultiColumnBlobs`.
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionFileStoreScan.java:
##########
@@ -281,6 +286,29 @@ static List<ManifestEntry> pruneByReadType(
if (anchor != null && !kept.contains(anchor)) {
kept.add(anchor);
}
+ // Blob and vector-store files may each cover only a sub-range of
their group. If the kept
+ // files are all dedicated files that together do not cover the
requested rows (the whole
+ // group when there is no row-range pushdown), the reader would derive
the range from those
+ // sub-ranges and drop the rows outside them, so keep the full-range
anchor. When they
+ // already cover the requested rows, the anchor is unnecessary and
would only read an extra
+ // file.
+ if (anchor == null
+ && !kept.isEmpty()
+ && kept.stream()
+ .allMatch(
+ e ->
+ isBlobFile(e.file().fileName())
+ ||
isVectorStoreFile(e.file().fileName()))) {
+ ManifestEntry fullRangeAnchor = retrieveAnchorFile(group,
ManifestEntry::file);
+ Range fullRange = fullRangeAnchor.file().nonNullRowIdRange();
+ List<Range> requested =
+ rowRangeIndex == null
+ ? Collections.singletonList(fullRange)
+ : rowRangeIndex.intersectedRanges(fullRange.from,
fullRange.to);
+ if (!coversRanges(kept, requested)) {
+ kept.add(fullRangeAnchor);
Review Comment:
Fixed. `DataEvolutionVectorReadPlanner` now plans explicit ranges when a
projected vector store covers only a sub-range of its group, so the uncovered
rows are NULL-filled through the range-aware path instead of falling to the
sequential reader that rejects the mismatched row count. Added
`VectorStoreTableTest#testPartialUpdateWithMissingVectorRangeWithoutDeletionVectors`
(no deletion vectors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
