JingsongLi commented on code in PR #10792:
URL: https://github.com/apache/paimon/pull/10792#discussion_r4369433749
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java:
##########
@@ -1620,10 +1685,72 @@ static long selectedRowCount(DataFileMeta file,
List<Range> rowRanges) {
.sum();
}
- private static long selectedRowCount(
+ @VisibleForTesting
+ static long selectedRowCount(
DataFileMeta file,
@Nullable List<Range> rowRanges,
- @Nullable FileIndexResult fileIndexResult) {
+ @Nullable FileIndexResult fileIndexResult,
+ @Nullable DeletionVector deletionVector,
+ @Nullable Range deletionVectorRange) {
+ if (deletionVector != null && !deletionVector.isEmpty()) {
+ RoaringBitmap32 selection = file.toFileSelection(rowRanges);
+ if (fileIndexResult instanceof BitmapIndexResult) {
+ RoaringBitmap32 indexSelection = ((BitmapIndexResult)
fileIndexResult).get();
+ selection =
+ selection == null
+ ? indexSelection.clone()
+ : RoaringBitmap32.and(selection,
indexSelection);
+ }
+ long rangeStart =
+ deletionVectorRange == null
+ ? file.nonNullFirstRowId()
+ : deletionVectorRange.from;
+ if (selection == null) {
+ if (deletionVectorRange != null
+ &&
deletionVectorRange.equals(file.nonNullRowIdRange())) {
+ return file.rowCount() - deletionVector.getCardinality();
Review Comment:
[P2] Base the sidecar decision on the selection actually passed to the reader
Subtracting the DV here changes the estimate without narrowing the format
reader's selection. `createFileReader` still builds
`FormatReaderContext.selection` only from `rowRanges` and the bitmap file-index
result; `ApplyDeletionVectorReader` filters deleted rows after the underlying
rows have been read.
With no bitmap selection, a 20,000-row file with 19,000 deleted rows now
passes the default 4,096-row / 5% thresholds and selects `.row`, but the row
reader still reads and decodes all 20,000 rows. I reproduced this with both
32-bit and 64-bit DVs; the baseline selects Parquet for the same inputs. This
lets dense scans bypass the sparse-sidecar safeguards and switch to full
row-store decoding, which is especially costly when projecting a few columns
from wide rows.
Please either push the DV into the actual reader selection and reuse that
selection for the heuristic, or retain the original estimate until that
pushdown is available. An end-to-end test should verify the selection or
physical rows/blocks read, rather than just the live-row count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
