Caideyipi commented on code in PR #18918:
URL: https://github.com/apache/iotdb/pull/18918#discussion_r4323674143
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java:
##########
@@ -137,7 +137,18 @@ public synchronized boolean reportStorageGroupStatus(
totalStorageGroupMemCost,
REJECT_THRESHOLD);
rejected = true;
- if (chooseMemTablesToMarkFlush(tsFileProcessor)) {
+ boolean currentProcessorSelected =
chooseMemTablesToMarkFlush(tsFileProcessor);
+ if (!currentProcessorSelected
+ && tsFileProcessor != null
+ && tsFileProcessor.getWorkMemTable() != null) {
+ // The writer holds its region's write lock. Waiting for another
processor in that region
+ // to flush would prevent the flush submitter from acquiring the same
lock. Let this writer
+ // finish and flush its own memtable, subject to the hard memory limit
below.
+ tsFileProcessor.setWorkMemTableShouldFlush();
+ flushTaskSubmitThreadPool.submit(tsFileProcessor::submitAFlushTask);
+ currentProcessorSelected = true;
+ }
+ if (currentProcessorSelected) {
Review Comment:
Pushed ab56b846a85090401530c43adaa6872d6ade1dc6 with the RPC capacity fix
and protection against repeated flush submission.
The fallback is retained: the inserting thread holds the region write lock,
so waiting for another processor's flush can prevent that flush submitter from
acquiring the same lock. Both the normal selection and fallback now reuse an
already marked memtable's pending flush instead of enqueueing another task on
every report. A replacement memtable can still be scheduled. Tests cover
small/first-tablet memtables, repeated reports, normal largest-first selection,
rejection recovery, and rejection at and above the hard limit.
This does **not** establish that the small-file risk is eliminated.
DirectFlushPolicy still closes the selected TsFileProcessor, and legitimately
selecting a small current memtable can still produce a small file. The existing
close-future guards make repeated close requests idempotent, but do not remove
that policy tradeoff. A matched old/new workload comparison of flush counts,
memtable sizes, file sizes, and write progress is still required; I am leaving
this discussion unresolved.
Separately, incremental RPC writes now only ensure capacity; both framed
transports make their shrink decision using the completed frame size. The
optional-growth fallback to the required capacity and failure accounting remain
covered. Against the original PR, the strengthened buffer suite reproduced 3
failing tests and SystemInfoTest reproduced 4; after the fix, service-rpc
verify passed all 108 English tests, the Chinese buffer suite passed 19, and
SystemInfoTest plus TsFileProcessorTest passed 36. Spotless, Checkstyle, and
the verify lifecycle checks passed.
I also rechecked REPORT.md's raw receiver logs: 6,833 / 4,276 / 11,478
distinct failures; all are read-buffer growth failures except 21 initial 1 KiB
read-buffer allocation failures on node 46. Startup logs show a shared RPC
budget of 858,993,459 bytes. A controlled read-budget rejection/release/retry
test passes with the buffer and memory-account implementations from before the
current branch's last six commits as well as the original PR. This supports
budget exhaustion as the immediate mechanism, but neither that test nor the
logs establish which change caused the workload's pressure. The independently
reproduced write shrink/grow regression is fixed; eliminating all workload
read-allocation failures remains unverified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]