chrisdutz commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4537103902
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,38 @@ public MemTableFlushTask(
writer.getFile().getName());
}
+ /**
+ * The encoded chunk buffers grow by doubling, so their capacity can reach
twice the data they
+ * hold.
+ */
+ private static final int CHUNK_BUFFER_GROWTH_FACTOR = 2;
+
+ /**
+ * Temporary memory of a flush: the bounded IO queue holds up to {@code
ioTaskQueueSize} encoded
+ * chunks of average size. With no more series than queue slots the queue
can't hold more than the
+ * memtable, so the raw share is capped at {@code memSize}; without the cap
a memtable of 3 series
+ * reserved 3.3 times its size, which can exceed the whole MemTable budget
and make the reject
+ * threshold negative. The share is doubled for the growth slack of the
chunk buffers.
+ */
+ static long estimateTemporaryMemory(
+ long memSize, int seriesNumber, int ioTaskQueueSize, boolean
encodingFasterThanIo) {
+ if (!encodingFasterThanIo || seriesNumber == 0) {
+ return 0;
+ }
+ long queuedShare =
+ seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber *
ioTaskQueueSize;
+ return queuedShare * CHUNK_BUFFER_GROWTH_FACTOR;
Review Comment:
Thanks, confirmed: with your INT32 case I get the same 1,048,912 bytes of
queued buffers against a reservation of 984,000.
To see how far this goes, I measured the encoded size against `memSize()`
for every data type and encoding with random full-range values (PLAIN time
column). Variable-length integers are the worst case: about 1.08 for PLAIN and
ZIGZAG `INT32` and 1.09 for ZIGZAG `INT64`, with a bound of 1.25 for the values
themselves (5 bytes for an `int`, 10 for a `long`). All other encodings stay
within about 6% (RLBE is the highest), and `TEXT` and `BOOLEAN` with RLE stay
below `memSize()`. Page headers are included in these numbers.
Since the buffers' capacity can reach twice their content, the estimate now
multiplies the queued share by 2.5 (2 × 1.25) instead of 2, still capped at the
memtable size (de0924b452). Your INT32 case (one series, 41,000 rows,
`Integer.MAX_VALUE - i`, PLAIN, uncompressed) is now part of
`testReservationCoversQueuedChunkBuffersWithStalledIo`; it fails with the
factor 2 and passes with 2.5.
I also considered a per-series bound from the encoders' maximum item sizes,
which would be tighter, but it means walking the memtable's schemas for the
estimate. I kept the constant for now; happy to switch if you prefer that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]