JingsongLi commented on code in PR #10959:
URL: https://github.com/apache/paimon/pull/10959#discussion_r4451752442


##########
paimon-core/src/main/java/org/apache/paimon/operation/RawFileSplitRead.java:
##########
@@ -146,6 +150,13 @@ public SplitRead<InternalRow> withReadType(RowType 
readRowType) {
     public RawFileSplitRead withFilter(Predicate predicate) {
         if (predicate != null) {
             this.filters = splitAnd(predicate);
+            if (rowTrackingEnabled) {
+                // Row ids and sequence numbers may only be in the file meta, 
not in the file.
+                List<Predicate> pushed =
+                        excludePredicateWithFields(filters, 
ROW_TRACKING_FIELD_NAMES);

Review Comment:
   [P2] Mark predicates referencing any tracking field as residual
   
   `excludePredicateWithFields` delegates to `containsFields`, which uses 
`fields.containsAll(leafPredicate.fieldNames())`. A predicate with both a 
user-column input and a tracking-field input therefore survives this exclusion, 
leaving `hasResidualFilters` false. This leaves a gap in the metadata-filter 
fix.
   
   I reproduced this on the current head through the table API: write three 
rows with `s = 'abc'` (row IDs 0, 1, 2), add a nullable column, then read with 
`SUBSTRING(s, _ROW_ID, 1) = 'b'`, `executeFilter()`, and `withLimit(1)`. The 
result is `[]` instead of the row with ID 2. Both the no-ALTER control and the 
no-limit control return the expected row. `SubstringTransform` supports BIGINT 
position inputs, so this is a valid predicate.
   
   For the old-schema file, `SchemaEvolutionUtil.devolveFilters` drops the 
transform, leaving `FormatReaderMapping` with no data filters. Since the 
predicate was not recognized as residual here, the limit is still passed to 
`FileIndexEvaluator`, which selects the first row before the original predicate 
runs.
   
   Please exclude a predicate when any referenced field belongs to 
`ROW_TRACKING_FIELD_NAMES`, using the same criterion on the scan side, and 
cover the mixed-input transform + schema change + limit combination. Changing 
the exclusion check to test for an intersection makes the reproducer pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to