danny0405 commented on code in PR #20654:
URL: https://github.com/apache/hudi/pull/20654#discussion_r4561149881


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/RecordIterators.java:
##########
@@ -157,6 +155,21 @@ public static ClosableIterator<RowData> 
getParquetRecordIterator(
     }
   }
 
+  static FilterPredicate createFilterPredicate(MessageType fileSchema, 
List<Predicate> predicates, FilterPredicate configuredFilter) {
+    FilterPredicate filterPredicate = configuredFilter;

Review Comment:
   [P2] Validate the configured predicate against the physical schema too
   
   The new guard only covers `predicates`; `configuredFilter` is copied 
straight into the result. If the Hadoop configuration contains 
`ParquetInputFormat.FILTER_PREDICATE = eq(binaryColumn("uuid"), "k")`, 
`getFilterPredicate(conf)` passes it here even for a native delete file whose 
schema contains only `_hoodie_record_key` and ordering fields. With an empty 
query-predicate list this method still returns that filter, and Parquet can 
discard every delete row group because `uuid` is absent, leaving the original 
tombstone-loss issue reachable through this supported configuration path.
   
   Could we apply the same physical-column validation to the configured 
predicate (conservatively omit the whole expression when a referenced column is 
absent), or explicitly prevent configured predicates from reaching native 
delete scans? Please add a regression that sets `FILTER_PREDICATE` in the 
configuration and verifies the tombstone survives. I reproduced the 
predicate-retention/row-group-drop behavior with an isolated probe of this 
helper's logic against Parquet 1.15.2; I have not run an end-to-end Flink test 
for this configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to