JingsongLi commented on code in PR #8116:
URL: https://github.com/apache/paimon/pull/8116#discussion_r3445704831
##########
paimon-core/src/main/java/org/apache/paimon/table/source/KeyValueTableRead.java:
##########
@@ -129,11 +140,36 @@ public InnerTableRead withTopN(TopN topN) {
@Override
public InnerTableRead withLimit(int limit) {
- initialized().forEach(r -> r.withLimit(limit));
this.limit = limit;
+ refreshReaderConfig();
return this;
}
+ @Override
+ public RecordReader<InternalRow> createReader(List<Split> splits) throws
IOException {
+ if (limit != null && limit > 0 && splits.size() > 1) {
+ boolean previousApplyMergeReadLimit = applyMergeReadLimit;
+ applyMergeReadLimit = false;
+ refreshReaderConfig();
+ try {
+ return
LimitRecordReader.limit(createConcatenatedReader(splits), limit);
+ } finally {
+ applyMergeReadLimit = previousApplyMergeReadLimit;
+ refreshReaderConfig();
+ }
+ }
+ return createConcatenatedReader(splits);
Review Comment:
This only wraps the concatenated reader when there are multiple splits, but
`MergeFileSplitRead.effectiveReadLimit()` can also return `null` for a single
split (for example with a non-primary-key/value filter, `forceKeepDelete`,
partial-update, aggregate, or deletion vectors). In those cases
`table.newRead().withLimit(10).createReader(plan.splits())` no longer enforces
the read limit at all when the plan has one split; the new
`MergeFileSplitReadTest#testWithLimitDisabledByNonPrimaryKeyFilter` even shows
the lower-level reader returning all rows once the merge-read optimization is
disabled. Please keep a global `LimitRecordReader` fallback at the table-read
level whenever the per-split merge-read limit is not applied, and add a
single-split + non-PK filter regression test through `KeyValueTableRead`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
