JackieTien97 commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4535398465


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,38 @@ public MemTableFlushTask(
         writer.getFile().getName());
   }
 
+  /**
+   * The encoded chunk buffers grow by doubling, so their capacity can reach 
twice the data they
+   * hold.
+   */
+  private static final int CHUNK_BUFFER_GROWTH_FACTOR = 2;
+
+  /**
+   * Temporary memory of a flush: the bounded IO queue holds up to {@code 
ioTaskQueueSize} encoded
+   * chunks of average size. With no more series than queue slots the queue 
can't hold more than the
+   * memtable, so the raw share is capped at {@code memSize}; without the cap 
a memtable of 3 series
+   * reserved 3.3 times its size, which can exceed the whole MemTable budget 
and make the reject
+   * threshold negative. The share is doubled for the growth slack of the 
chunk buffers.
+   */
+  static long estimateTemporaryMemory(
+      long memSize, int seriesNumber, int ioTaskQueueSize, boolean 
encodingFasterThanIo) {
+    if (!encodingFasterThanIo || seriesNumber == 0) {
+      return 0;
+    }
+    long queuedShare =
+        seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber * 
ioTaskQueueSize;
+    return queuedShare * CHUNK_BUFFER_GROWTH_FACTOR;

Review Comment:
   [P2] Account for encoding expansion before applying the buffer growth factor
   
   The 2× growth bound applies to the encoded byte count, but `memSize()` 
counts raw values. For example, PLAIN-encoded `INT32` values can occupy 5 bytes 
each while `memSize()` counts only 4, so doubling the raw share can still 
under-reserve the queued buffers.
   
   I reproduced this on `d458e7be55` using the full `MemTableFlushTask` 
pipeline with I/O stalled at `startChunkGroup`: one device, one non-aligned 
`INT32` series, 41,000 rows, timestamps `i`, values `Integer.MAX_VALUE - i`, 
PLAIN time/value encoders, and UNCOMPRESSED data. With the default queue/chunk 
settings, the queue contains one chunk writer:
   
   | Measurement | Bytes |
   | --- | ---: |
   | `memSize()` | 492,000 |
   | Retained buffers, measured with this PR's `retainedArrayBytes` helper | 
1,048,912 |
   | Actual reservation with this PR | 984,000 |
   | Actual reservation with the base flush implementation | 4,920,000 |
   
   The same buffer-coverage assertion fails with the PR implementation and 
passes with the base implementation. This leaves at least 64,912 bytes 
unaccounted for when reducing the MemTable budget and computing the 
write-rejection threshold. The existing 18 targeted tests pass, but the new 
stalled-I/O cases only exercise DOUBLE values.
   
   Could we account for encoding expansion and page overhead before applying 
the capacity-growth allowance, and add this INT32 case to the stalled-I/O 
regression test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to