JingsongLi commented on code in PR #8116:
URL: https://github.com/apache/paimon/pull/8116#discussion_r3519628396
##########
paimon-core/src/main/java/org/apache/paimon/table/source/KeyValueTableRead.java:
##########
@@ -134,6 +136,31 @@ public InnerTableRead withLimit(int limit) {
return this;
}
+ @Override
+ public RecordReader<InternalRow> createReader(List<Split> splits) throws
IOException {
+ return LimitRecordReader.limit(super.createReader(splits), limit);
+ }
+
+ @Override
+ public RecordReader<InternalRow> createReader(Split split) throws
IOException {
+ // Query-auth filters run after split read; skip merge-read limit for
those splits.
+ if (!hasLateAppliedRowFilter(split)) {
+ return super.createReader(split);
Review Comment:
This still drops the limit on the public single-split API whenever the
merge-read limit is disabled for reasons other than query auth. For example,
`table.newRead().withFilter(nonPrimaryKeyFilter).withLimit(10).createReader(split)`
reaches this branch, `MergeFileSplitRead.effectiveReadLimit()` returns `null`
because of the non-PK filter, and there is no outer `LimitRecordReader`, so the
reader can return all matching rows. The same applies to other disabled cases
such as `forceKeepDelete`, partial-update/aggregation, or deletion vectors.
Please apply the table-level limit fallback for any split whose lower-level
read cannot apply the merge-read limit, not only for `QueryAuthSplit`, and add
a `createReader(Split)` regression for a non-PK filter/disabled-limit case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
