JackieTien97 commented on code in PR #19108:
URL: https://github.com/apache/iotdb/pull/19108#discussion_r4535398465
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/flush/MemTableFlushTask.java:
##########
@@ -141,15 +141,38 @@ public MemTableFlushTask(
writer.getFile().getName());
}
+ /**
+ * The encoded chunk buffers grow by doubling, so their capacity can reach
twice the data they
+ * hold.
+ */
+ private static final int CHUNK_BUFFER_GROWTH_FACTOR = 2;
+
+ /**
+ * Temporary memory of a flush: the bounded IO queue holds up to {@code
ioTaskQueueSize} encoded
+ * chunks of average size. With no more series than queue slots the queue
can't hold more than the
+ * memtable, so the raw share is capped at {@code memSize}; without the cap
a memtable of 3 series
+ * reserved 3.3 times its size, which can exceed the whole MemTable budget
and make the reject
+ * threshold negative. The share is doubled for the growth slack of the
chunk buffers.
+ */
+ static long estimateTemporaryMemory(
+ long memSize, int seriesNumber, int ioTaskQueueSize, boolean
encodingFasterThanIo) {
+ if (!encodingFasterThanIo || seriesNumber == 0) {
+ return 0;
+ }
+ long queuedShare =
+ seriesNumber <= ioTaskQueueSize ? memSize : memSize / seriesNumber *
ioTaskQueueSize;
+ return queuedShare * CHUNK_BUFFER_GROWTH_FACTOR;
Review Comment:
[P2] Account for encoding expansion before applying the buffer growth factor
The 2× growth bound applies to the encoded byte count, but `memSize()`
counts raw values. For example, PLAIN-encoded `INT32` values can occupy 5 bytes
each while `memSize()` counts only 4, so doubling the raw share can still
under-reserve the queued buffers.
I reproduced this on `d458e7be55` using the full `MemTableFlushTask`
pipeline with I/O stalled at `startChunkGroup`: one device, one non-aligned
`INT32` series, 41,000 rows, timestamps `i`, values `Integer.MAX_VALUE - i`,
PLAIN time/value encoders, and UNCOMPRESSED data. With the default queue/chunk
settings, the queue contains one chunk writer:
| Measurement | Bytes |
| --- | ---: |
| `memSize()` | 492,000 |
| Retained buffers, measured with this PR's `retainedArrayBytes` helper |
1,048,912 |
| Actual reservation with this PR | 984,000 |
| Actual reservation with the base flush implementation | 4,920,000 |
The same buffer-coverage assertion fails with the PR implementation and
passes with the base implementation. This leaves at least 64,912 bytes
unaccounted for when reducing the MemTable budget and computing the
write-rejection threshold. The existing 18 targeted tests pass, but the new
stalled-I/O cases only exercise DOUBLE values.
Could we account for encoding expansion and page overhead before applying
the capacity-growth allowance, and add this INT32 case to the stalled-I/O
regression test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]