danny0405 commented on code in PR #8051:
URL: https://github.com/apache/hudi/pull/8051#discussion_r1127309462
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -213,24 +213,17 @@ public void setFilters(List<ResolvedExpression> filters) {
*/
@Nullable
private Set<String> candidateFilesInMetadataTable(FileStatus[]
allFileStatus) {
- // NOTE: Data Skipping is only effective when it references columns that
are indexed w/in
- // the Column Stats Index (CSI). Following cases could not be
effectively handled by Data Skipping:
- // - Expressions on top-level column's fields (ie, for ex filters
like "struct.field > 0", since
- // CSI only contains stats for top-level columns, in this case
for "struct")
- // - Any expression not directly referencing top-level column
(for ex, sub-queries, since there's
- // nothing CSI in particular could be applied for)
- if (!metadataConfig.enabled() || !dataSkippingEnabled) {
- validateConfig();
- return null;
+ // initialize data skipper if it's not initialized yet
+ if (this.dataPrunerOpt == null) {
+ DataPruner dataPruner = initializeDataPruner();
Review Comment:
Can we init the pruner in FileIndex constructor, make sure the `DataPruner`
is serializabel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
