Caideyipi commented on code in PR #18918:
URL: https://github.com/apache/iotdb/pull/18918#discussion_r4323674143


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java:
##########
@@ -137,7 +137,18 @@ public synchronized boolean reportStorageGroupStatus(
           totalStorageGroupMemCost,
           REJECT_THRESHOLD);
       rejected = true;
-      if (chooseMemTablesToMarkFlush(tsFileProcessor)) {
+      boolean currentProcessorSelected = 
chooseMemTablesToMarkFlush(tsFileProcessor);
+      if (!currentProcessorSelected
+          && tsFileProcessor != null
+          && tsFileProcessor.getWorkMemTable() != null) {
+        // The writer holds its region's write lock. Waiting for another 
processor in that region
+        // to flush would prevent the flush submitter from acquiring the same 
lock. Let this writer
+        // finish and flush its own memtable, subject to the hard memory limit 
below.
+        tsFileProcessor.setWorkMemTableShouldFlush();
+        flushTaskSubmitThreadPool.submit(tsFileProcessor::submitAFlushTask);
+        currentProcessorSelected = true;
+      }
+      if (currentProcessorSelected) {

Review Comment:
   Pushed ab56b846a85090401530c43adaa6872d6ade1dc6 with the RPC capacity fix 
and protection against repeated flush submission.
   
   The fallback is retained: the inserting thread holds the region write lock, 
so waiting for another processor's flush can prevent that flush submitter from 
acquiring the same lock. Both the normal selection and fallback now reuse an 
already marked memtable's pending flush instead of enqueueing another task on 
every report. A replacement memtable can still be scheduled. Tests cover 
small/first-tablet memtables, repeated reports, normal largest-first selection, 
rejection recovery, and rejection at and above the hard limit.
   
   This does **not** establish that the small-file risk is eliminated. 
DirectFlushPolicy still closes the selected TsFileProcessor, and legitimately 
selecting a small current memtable can still produce a small file. The existing 
close-future guards make repeated close requests idempotent, but do not remove 
that policy tradeoff. A matched old/new workload comparison of flush counts, 
memtable sizes, file sizes, and write progress is still required; I am leaving 
this discussion unresolved.
   
   Separately, incremental RPC writes now only ensure capacity; both framed 
transports make their shrink decision using the completed frame size. The 
optional-growth fallback to the required capacity and failure accounting remain 
covered. Against the original PR, the strengthened buffer suite reproduced 3 
failing tests and SystemInfoTest reproduced 4; after the fix, service-rpc 
verify passed all 108 English tests, the Chinese buffer suite passed 19, and 
SystemInfoTest plus TsFileProcessorTest passed 36. Spotless, Checkstyle, and 
the verify lifecycle checks passed.
   
   I also rechecked REPORT.md's raw receiver logs: 6,833 / 4,276 / 11,478 
distinct failures; all are read-buffer growth failures except 21 initial 1 KiB 
read-buffer allocation failures on node 46. Startup logs show a shared RPC 
budget of 858,993,459 bytes. A controlled read-budget rejection/release/retry 
test passes with the buffer and memory-account implementations from before the 
current branch's last six commits as well as the original PR. This supports 
budget exhaustion as the immediate mechanism, but neither that test nor the 
logs establish which change caused the workload's pressure. The independently 
reproduced write shrink/grow regression is fixed; eliminating all workload 
read-allocation failures remains unverified.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to