xicm commented on code in PR #7226:
URL: https://github.com/apache/hudi/pull/7226#discussion_r1027564734
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java:
##########
@@ -367,9 +368,19 @@ public int getPartition(Object key) {
/**
* Obtains the average record size based on records written during previous
commits. Used for estimating how many
* records pack into one file.
+ * Respect user setting by following the precedence as below
+ * 1) if user sets a value, then use it as is
+ * 2) if user not setting it, infer from timeline commit metadata
+ * 3) if timeline is empty, use a default (current: 1024)
*/
protected static long averageBytesPerRecord(HoodieTimeline commitTimeline,
HoodieWriteConfig hoodieWriteConfig) {
+ long defaultAvgSize =
Integer.parseInt(HoodieCompactionConfig.COPY_ON_WRITE_RECORD_SIZE_ESTIMATE.defaultValue());
long avgSize = hoodieWriteConfig.getCopyOnWriteRecordSizeEstimate();
+
+ if (avgSize != defaultAvgSize) {
Review Comment:
@danny0405 This pr makes user-provided avg value first. We can't tell the
value is default or user-provided here, so I suppose if the avg size is not
default, it is user provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
