danny0405 commented on code in PR #18843:
URL: https://github.com/apache/hudi/pull/18843#discussion_r3308684436
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -672,17 +699,29 @@ private void bufferDelete(HoodieRecord<T> hoodieRecord) {
/**
* Checks if the number of records have reached the set threshold and then
flushes the records to disk.
+ *
+ * <p>{@code bufferedRecord} is the record that was just appended to {@link
#recordList} by
+ * {@link #writeToBuffer} (or {@code null} for delete/ignored windows where
{@code recordList}
+ * did not grow). Sizing this object — rather than the incoming pre-{@code
prepareRecord}
+ * record — keeps {@link #averageRecordSize} aligned with what is actually
retained in heap.
+ * The incoming record's payload is typically still compact/deflated; the
buffered record
+ * holds the fully-materialized Avro {@code IndexedRecord} with prepended
meta-fields, which
+ * is what {@code maxBlockSize} is meant to bound.
*/
- protected void flushToDiskIfRequired(HoodieRecord record, boolean
appendDeleteBlocks) {
- if (numberOfRecords >= (int) (maxBlockSize / averageRecordSize)
- || numberOfRecords % NUMBER_OF_RECORDS_TO_ESTIMATE_RECORD_SIZE == 0) {
- averageRecordSize = (long) (averageRecordSize * 0.8 +
sizeEstimator.sizeEstimate(record) * 0.2);
+ protected void flushToDiskIfRequired(HoodieRecord bufferedRecord, boolean
appendDeleteBlocks) {
+ if (bufferedRecord != null
Review Comment:
> for delete/ignored windows where {@code recordList} did not grow
This is true for `recordList` but the deletes are cached in
`recordsToDeleteWithPositions` which also takes memry, we should not ignore the
delete payload IMO.
Assumps the deletes and inserts/updates are evenly distributed, taking the
deletes while estimating the buffer size by sampling can reflect the real
buffer size because the `numberOfRecords` actually take the deletes cnt too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
