chrisdutz commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4537103902


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,38 @@ public MemTableFlushTask(
         writer.getFile().getName());
   }
 
+  /**
+   * The encoded chunk buffers grow by doubling, so their capacity can reach 
twice the data they
+   * hold.
+   */
+  private static final int CHUNK_BUFFER_GROWTH_FACTOR = 2;
+
+  /**
+   * Temporary memory of a flush: the bounded IO queue holds up to {@code 
ioTaskQueueSize} encoded
+   * chunks of average size. With no more series than queue slots the queue 
can't hold more than the
+   * memtable, so the raw share is capped at {@code memSize}; without the cap 
a memtable of 3 series
+   * reserved 3.3 times its size, which can exceed the whole MemTable budget 
and make the reject
+   * threshold negative. The share is doubled for the growth slack of the 
chunk buffers.
+   */
+  static long estimateTemporaryMemory(
+      long memSize, int seriesNumber, int ioTaskQueueSize, boolean 
encodingFasterThanIo) {
+    if (!encodingFasterThanIo || seriesNumber == 0) {
+      return 0;
+    }
+    long queuedShare =
+        seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber * 
ioTaskQueueSize;
+    return queuedShare * CHUNK_BUFFER_GROWTH_FACTOR;

Review Comment:
   Thanks, confirmed: with your INT32 case I get the same 1,048,912 bytes of 
queued buffers against a reservation of 984,000.
   
   To see how far this goes, I measured the encoded size against `memSize()` 
for every data type and encoding with random full-range values (PLAIN time 
column). Variable-length integers are the worst case: about 1.08 for PLAIN and 
ZIGZAG `INT32` and 1.09 for ZIGZAG `INT64`, with a bound of 1.25 for the values 
themselves (5 bytes for an `int`, 10 for a `long`). All other encodings stay 
within about 6% (RLBE is the highest), and `TEXT` and `BOOLEAN` with RLE stay 
below `memSize()`. Page headers are included in these numbers.
   
   Since the buffers' capacity can reach twice their content, the estimate now 
multiplies the queued share by 2.5 (2 × 1.25) instead of 2, still capped at the 
memtable size (de0924b452). Your INT32 case (one series, 41,000 rows, 
`Integer.MAX_VALUE - i`, PLAIN, uncompressed) is now part of 
`testReservationCoversQueuedChunkBuffersWithStalledIo`; it fails with the 
factor 2 and passes with 2.5.
   
   I also considered a per-series bound from the encoders' maximum item sizes, 
which would be tighter, but it means walking the memtable's schemas for the 
estimate. I kept the constant for now; happy to switch if you prefer that.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to