cshuo commented on code in PR #17490:
URL: https://github.com/apache/hudi/pull/17490#discussion_r2614020267
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -417,6 +417,18 @@ private FlinkOptions() {
.withDescription("Enables data-skipping allowing queries to leverage
indexes to reduce the search space by "
+ "skipping over files");
+ @AdvancedConfig
+ public static final ConfigOption<Integer>
READ_DATA_SKIPPING_RLI_KEYS_MAX_NUM = ConfigOptions
+ .key("read.data.skipping.rli.keys.max.num")
+ .intType()
+ .defaultValue(8)
+ .withDescription("Record Level index statistics will be read from
metadata table (MDT) for data skipping optimization,\n"
+ + "and currently the index statistics are collected by a single
process. This config is used to constrain the maximum \n"
+ + " number of hoodie keys that can be read from MDT without
sacrificing any performance. If the number of hoodie keys from query\n"
+ + "predicate is greater than the maximum value, the query will
fallback to skip the record level index filtering.\n"
+ + "E.g., given query: SELECT * FROM T WHERE `uuid` IN
(1,2,3,4,5,6,7,8,9), the number of hoodie keys is 9, and\n"
+ + "the maximum value is 8, so the source will not perform record
level index filtering.");
Review Comment:
https://github.com/apache/hudi/issues/17582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
