wwj6591812 commented on code in PR #8116:
URL: https://github.com/apache/paimon/pull/8116#discussion_r3445758325
##########
paimon-core/src/main/java/org/apache/paimon/table/source/KeyValueTableRead.java:
##########
@@ -129,11 +140,36 @@ public InnerTableRead withTopN(TopN topN) {
@Override
public InnerTableRead withLimit(int limit) {
- initialized().forEach(r -> r.withLimit(limit));
this.limit = limit;
+ refreshReaderConfig();
return this;
}
+ @Override
+ public RecordReader<InternalRow> createReader(List<Split> splits) throws
IOException {
+ if (limit != null && limit > 0 && splits.size() > 1) {
+ boolean previousApplyMergeReadLimit = applyMergeReadLimit;
+ applyMergeReadLimit = false;
+ refreshReaderConfig();
+ try {
+ return
LimitRecordReader.limit(createConcatenatedReader(splits), limit);
+ } finally {
+ applyMergeReadLimit = previousApplyMergeReadLimit;
+ refreshReaderConfig();
+ }
+ }
+ return createConcatenatedReader(splits);
Review Comment:
Thanks for the review, @JingsongLi .
You're right — the global LimitRecordReader wrapper was only applied for
multi-split plans, so when a single-split plan hit a case where
MergeFileSplitRead.effectiveReadLimit() returns null (e.g. non-PK filter),
withLimit(n) no longer enforced the limit at the table-read level.
I've added SplitRead#isMergeReadLimitActive() and updated
KeyValueTableRead#createReader(List) to apply a table-level LimitRecordReader
fallback whenever merge-read limit is not active (including single-split +
non-PK filter). I also added
testReadWithLimitThroughTableReadPathWithNonPrimaryKeyFilter in
PrimaryKeySimpleTableTest to cover this through the full KeyValueTableRead
path. Please take another look when you have a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
