zhuanshenbsj1 commented on code in PR #10328:
URL: https://github.com/apache/hudi/pull/10328#discussion_r1555497026
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -329,6 +329,14 @@ private FlinkOptions() {
.withDescription("Whether to skip clustering instants to avoid reading
base files of clustering operations for streaming read "
+ "to improve read performance.");
+ // this option is experimental
+ public static final ConfigOption<Boolean>
READ_STREAMING_SKIP_INSERT_OVERWRITE = ConfigOptions
+ .key("read.streaming.skip_insertoverwrite")
+ .booleanType()
+ .defaultValue(false)
Review Comment:
This is different from clustering and compaction. Skipping clustering and
compaction will not cause inconsistency in the upstream and downstream data in
the streaming reading scenario. By default, the upstream and downstream data
should be consistent. Developers must make their own decisions knowing that
this will lead to inconsistency in the upstream and downstream data， and then
enable this feature as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
