danny0405 commented on code in PR #8437:
URL: https://github.com/apache/hudi/pull/8437#discussion_r1225147219
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/RecordIterators.java:
##########
@@ -80,12 +104,42 @@ public static ClosableIterator<RowData>
getParquetRecordIterator(
batchSize,
path,
splitStart,
- splitLength));
+ splitLength,
+ filterPredicate,
+ recordFilter));
if (castProjection.isPresent()) {
return new SchemaEvolvedRecordIterator(itr, castProjection.get());
} else {
return itr;
}
}
}
+
+ private static FilterPredicate getFilterPredicate(Configuration
configuration) {
+ try {
+ return SerializationUtil.readObjectFromConfAsBase64(FILTER_PREDICATE,
configuration);
+ } catch (IOException e) {
Review Comment:
Not sure how the user can config the predicate in configuration, and what
the `UNBOUND_RECORD_FILTER` is used for ? Because Hudi may has merging logic of
base parquet file and logs files, does these filtering utilities only work for
parquets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
