yanghua commented on a change in pull request #2785:
URL: https://github.com/apache/hudi/pull/2785#discussion_r610278459
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java
##########
@@ -159,8 +164,11 @@ public void processElement(I value, Context ctx,
Collector<O> out) throws Except
final BucketInfo bucketInfo;
final HoodieRecordLocation location;
- if (!partitionLoadState.contains(hoodieKey.getPartitionPath())) {
+ if (bootstrapIndex &&
!partitionLoadState.contains(hoodieKey.getPartitionPath())) {
// If the partition records are never loaded, load the records first.
+
+ // The dataset may be huge, thus the processing would block for long,
Review comment:
I did not question the necessity of this comment, just to see if it has
a more suitable position. Because when I see this comment, the method is
already executing. And your comment contains "disabled by default", but this
method does not accept a flag. Personally, I think a better position is placed
before the above if statement, where the execution switch is.
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java
##########
@@ -255,8 +264,14 @@ private FlinkOptions() {
public static final ConfigOption<Double> WRITE_BATCH_SIZE = ConfigOptions
.key("write.batch.size.MB")
.doubleType()
- .defaultValue(2D) // 2MB
- .withDescription("Batch buffer size in MB to flush data into the
underneath filesystem");
+ .defaultValue(64D) // 64MB
+ .withDescription("Batch buffer size in MB to flush data into the
underneath filesystem, default 64MB");
+
+ public static final ConfigOption<Integer> WRITE_LOG_BLOCK_SIZE =
ConfigOptions
+ .key("write.log_block.size")
Review comment:
IMO, between verbosity and following a uniform style, I think the latter
is more important and allows users to feel a consistent experience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
