HTHou commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4508754416
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,30 @@ public MemTableFlushTask(
writer.getFile().getName());
}
+ /**
+ * Temporary memory of a flush: the bounded IO queue holds up to {@code
ioTaskQueueSize} encoded
+ * series of average size. With no more series than queue slots the queue
can't hold more than the
+ * memtable, so the estimate is capped at {@code memSize}. Without the cap a
memtable of 3 series
+ * reserved 3.3 times its size, which can exceed the whole MemTable budget
and make the reject
+ * threshold negative.
+ */
+ static long estimateTemporaryMemory(
+ long memSize, int seriesNumber, int ioTaskQueueSize, boolean
encodingFasterThanIo) {
+ if (!encodingFasterThanIo || seriesNumber == 0) {
+ return 0;
+ }
+ return seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber
* ioTaskQueueSize;
Review Comment:
Could we leave some headroom for the encoded chunk buffers here? The I/O
queue holds chunk writers, and a single series can enqueue multiple chunks. The
writers' backing arrays can also grow larger than the raw data counted by
`memSize()`.
I reproduced this by pausing I/O until the queue filled, using one device
with nine `DOUBLE` series (`PLAIN`, `UNCOMPRESSED`, 60,000 points per series).
The queued writers' backing arrays alone occupied 9,406,224 bytes on both
revisions. The base reserved 9,600,000 bytes, while this PR reserved 8,640,000
bytes. That leaves 766,224 bytes unaccounted for when computing the write
rejection threshold.
Can we keep the reduction in over-reservation while accounting for those
buffers, and add a regression test that exercises the reservation with a
stalled I/O consumer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]