HTHou commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4508754416


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,30 @@ public MemTableFlushTask(
         writer.getFile().getName());
   }
 
+  /**
+   * Temporary memory of a flush: the bounded IO queue holds up to {@code 
ioTaskQueueSize} encoded
+   * series of average size. With no more series than queue slots the queue 
can't hold more than the
+   * memtable, so the estimate is capped at {@code memSize}. Without the cap a 
memtable of 3 series
+   * reserved 3.3 times its size, which can exceed the whole MemTable budget 
and make the reject
+   * threshold negative.
+   */
+  static long estimateTemporaryMemory(
+      long memSize, int seriesNumber, int ioTaskQueueSize, boolean 
encodingFasterThanIo) {
+    if (!encodingFasterThanIo || seriesNumber == 0) {
+      return 0;
+    }
+    return seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber 
* ioTaskQueueSize;

Review Comment:
   Could we leave some headroom for the encoded chunk buffers here? The I/O 
queue holds chunk writers, and a single series can enqueue multiple chunks. The 
writers' backing arrays can also grow larger than the raw data counted by 
`memSize()`.
   
   I reproduced this by pausing I/O until the queue filled, using one device 
with nine `DOUBLE` series (`PLAIN`, `UNCOMPRESSED`, 60,000 points per series). 
The queued writers' backing arrays alone occupied 9,406,224 bytes on both 
revisions. The base reserved 9,600,000 bytes, while this PR reserved 8,640,000 
bytes. That leaves 766,224 bytes unaccounted for when computing the write 
rejection threshold.
   
   Can we keep the reduction in over-reservation while accounting for those 
buffers, and add a regression test that exercises the reservation with a 
stalled I/O consumer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to