chrisdutz opened a new pull request, #19108:
URL: https://github.com/apache/iotdb/pull/19108
## Description
`MemTableFlushTask.syncFlushMemTable()` reserves temporary memory for a
flush and takes it out of
the MemTable budget (`SystemInfo.reserveTemporaryMemoryForFlushing`, since
#18848; before that
`applyTemporaryMemoryForFlushing` with the same estimate):
```
memSize / seriesNumber * io_task_queue_size_for_flushing
```
The bounded IO queue holds up to `io_task_queue_size_for_flushing` (default
10) encoded series of
average size. With fewer series than queue slots the formula estimates more
than the memtable
itself: a memtable with 3 series reserves 10/3 = 3.3 times its size. With a
small MemTable budget
a single flush can then reserve more than the whole budget, so
`FLUSH_THRESHOLD` and
`REJECT_THRESHOLD` become negative and the node rejects writes until the
flush is done.
Measured with the same code in a downstream fork (standalone DataNode, 147.9
MB MemTable budget),
writing to one device with 3 series (flushes of about 52 MB):
| | largest reservation | lowest reject threshold
|
|----------------------------|---------------------|-------------------------|
| before | 174.5 MB | -21.3 MB (rejecting)
|
| after | 52.4 MB | 76.4 MB
|
| before, 100 devices x 3 series (not affected) | 1.7 MB | 116.9 MB |
The queue can never hold more than the memtable, so the estimate is now
capped at `memSize`
when there are no more series than queue slots; with more series it is
unchanged.
The estimate is moved into a small static method
`estimateTemporaryMemory(...)` so that it can
be unit tested (`FlushMemoryEstimateTest`).
## Tests
- New `FlushMemoryEstimateTest` (fails before the change: 173.3 MB estimated
for a 52 MB memtable
with 3 series).
- `MemTableFlushTaskTest`, `TsFileProcessorTest`, `SystemInfoTest`,
`DataRegionTest`: 106 tests
pass; `spotless:check` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]