danny0405 commented on a change in pull request #2553:
URL: https://github.com/apache/hudi/pull/2553#discussion_r572824048
##########
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/StreamWriteFunction.java
##########
@@ -252,6 +270,41 @@ private void initWriteFunction() {
}
}
+ /**
+ * Tool to detect if to flush out the existing buffer.
+ * Sampling the record to compute the size with 0.01 percentage.
+ */
+ private static class BufferSizeDetector {
+ private final Random random = new Random(47);
+
+ private final double batchSizeBytes;
+
+ private long lastRecordSize = -1L;
+ private long totalSize = 0L;
+
+ BufferSizeDetector(double batchSizeMb) {
+ this.batchSizeBytes = batchSizeMb * 1024 * 1024;
Review comment:
I'm not planning to support unit in the config option value, so a key
with unit is more clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
