danny0405 commented on code in PR #6856:
URL: https://github.com/apache/hudi/pull/6856#discussion_r1095369349
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -297,19 +299,22 @@ private FlinkOptions() {
.key("read.streaming.skip_compaction")
.booleanType()
.defaultValue(false)// default read as batch
- .withDescription("Whether to skip compaction instants for streaming
read,\n"
- + "there are two cases that this option can be used to avoid reading
duplicates:\n"
- + "1) you are definitely sure that the consumer reads faster than
any compaction instants, "
- + "usually with delta time compaction strategy that is long enough,
for e.g, one week;\n"
+ .withDescription("Whether to skip compaction instants and avoid reading
compacted base files for streaming read to improve read performance.\n"
+ + "There are two cases that this option can be used to avoid reading
duplicates:\n"
+ + "1) you are definitely sure that the consumer reads [faster
than/completes before] any compaction instants "
+ + "when " + HoodieCompactionConfig.PRESERVE_COMMIT_METADATA.key() +
" is set to false.\n"
+ "2) changelog mode is enabled, this option is a solution to keep
data integrity");
// this option is experimental
public static final ConfigOption<Boolean> READ_STREAMING_SKIP_CLUSTERING =
ConfigOptions
.key("read.streaming.skip_clustering")
.booleanType()
.defaultValue(false)
- .withDescription("Whether to skip clustering instants for streaming
read,\n"
- + "to avoid reading duplicates");
+ .withDescription("Whether to skip clustering instants to avoid
reading base files of clustering operations for streaming read "
+ + "to improve read performance.\n"
+ + "This option toggled to true to avoid duplicates when: \n"
+ + "1) you are definitely sure that the consumer reads [faster
than/completes before] any clustering instants "
Review Comment:
We actually filter the clustering instants by files not rows, so same with
compaction, the ` HoodieCompactionConfig.PRESERVE_COMMIT_METADATA.key()` has no
effect here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
