hongzhi-gao opened a new pull request, #18848:
URL: https://github.com/apache/iotdb/pull/18848

   ## Description
   
   Write-memory accounting is spread across MemTables, TsFileProcessors and 
DataRegions, requiring callers to update several counters together. This change 
introduces named, categorized owner blocks that propagate usage to their 
accounting scope and tie rollback/release to the owner's lifecycle.
   
   ### Accounting and lifecycle
   
   - Add `WriteMemoryBlock` and `WriteMemoryManager` for MemTable arrays, 
Binary values, recovered totals, TsFile metadata and flush temporary 
reservations. Pending reservations count toward admission but become used 
MemTable memory only after admission succeeds. WAL failures roll back the 
admitted deltas.
   - Replace positional write-memory deltas with named fields across row, 
mixed-row and split-tablet paths. MemTable release closes its blocks and 
updates the region total once; query-held TVLists retain their existing 
resource ownership.
   - Preserve the WAL snapshot's aggregate memory field. Recovery places that 
aggregate in a separate category rather than guessing an array/Binary breakdown.
   - Scope flush reservations to actual worker lifetime: propagate worker 
failures, cancel and await both workers, clear temporary references, then 
release the reservation. Budget reload retains outstanding reservations.
   - Store each WAL queue entry's original charge and refund it on dequeue or 
interrupted enqueue.
   
   ### Array-pool accounting and hot paths
   
   Count idle arrays by actual shallow size under the existing per-type queue 
locks. A new `ReadOnlyMemoryBlock` exposes an exact snapshot to the existing 
memory-manager tree; readers acquire queue locks in a fixed order. This avoids 
an additional global atomic update on each array borrow/return. 
`WriteMemoryBlock` uses a monitor-guarded `long`; region totals remain atomic. 
Zero-delta writes skip reservation creation while retaining pressure checks.
   
   SQL, configured memory limits, file/WAL formats and the existing admission 
policy are unchanged. Accounting follows write ownership, not total JVM heap 
liveness. New exception messages are provided in both English and Chinese.
   
   ### Verification
   
   Executed remotely on Linux with JDK 21 (production compilation targets Java 
17), against the exact submitted source:
   
   - Spotless and focused Maven tests: **122 passed**, zero 
failures/errors/skips (7 Node Commons, 115 DataNode).
   - Full-reactor `mvn test-compile -DskipTests` and `mvn test-compile 
-DskipTests -P with-zh-locale`: passed. Packaging prerequisites were built with 
`mvn install -DskipTests -pl 
iotdb-core/datanode,iotdb-core/confignode,iotdb-client/cli -am`.
   - **4 lifecycle ITs passed**, using the repository's real 1C1D fixtures 
through JUnitCore, rather than Maven Failsafe. Coverage includes 63/64/65 array 
boundaries, sparse/new aligned columns, repeated flush/unsequence overwrite, 
oversized-WAL row/tablet rejection and retry, forced DataNode restart/recovery, 
and table writes split across devices/time partitions. Tests verify query 
results and live database write-memory metrics.
   - Targeted tests also cover concurrent reservations/snapshots, query-held 
TVList ownership, interrupted WAL enqueue, and flush failure/interruption 
cleanup.
   
   The focused Maven test selection was:
   
   ```sh
   mvn test -pl iotdb-core/node-commons,iotdb-core/datanode \
     
-Dtest=MemoryBlockTest,MemoryManagerTest,ReadOnlyMemoryBlockTest,WriteMemoryBlockTest,WriteMemoryLifecycleTest,MemTableMemoryBlockTest,TsFileProcessorTest,MemTableFlushTaskTest,PrimitiveArrayManagerTest,BooleanTVListTest,IntTVListTest,LongTVListTest,FloatTVListTest,DoubleTVListTest,BinaryTVListTest,AlignedTVListTest
 \
     -Dsurefire.failIfNoSpecifiedTests=false
   ```
   
   Initial environment failures were resolved before reporting these results: 
missing distribution ZIP prerequisites, and JMX port collisions during IT node 
startup. The unchanged ITs passed after rerunning with free ports.
   
   ### Performance observations
   
   Compared with `eb0e3f1f2080e34bf5593bfd1cf63b62ecdba2e0`, using the same 
workload/configuration and three interleaved runs per version:
   
   | Workload (4 writers) | Median throughput | P50 RPC latency | P99 RPC 
latency |
   | --- | ---: | ---: | ---: |
   | Non-aligned Tablet, 1024 rows | -0.24% | +3.88% | +0.10% |
   | Aligned single row | +0.50% | -0.73% | +3.19% |
   | Aligned Tablet, 1024 rows | +4.36% | -4.60% | -3.58% |
   
   Environment: shared Xeon Gold 6130 KVM host, 32 vCPUs, JDK 21, 
1C1D/SimpleConsensus, ASYNC WAL, memory control enabled, 2 GiB DataNode heap, 
array size 64, MemTable threshold 32 MiB, compaction and timed flush disabled. 
Fixed per-writer warmup/measurement calls were 50k/100k for non-aligned Tablet, 
125k/250k for aligned rows, and 20k/40k for aligned Tablet. Each run checked 
acknowledged row counts through queries after a final FLUSH.
   
   These are controlled class-overlay comparisons, not two independent clean 
distribution builds. The throughput window excludes the final FLUSH; it does 
not measure synchronous durability throughput. Shared-host variability and 
existing QueryId waits affect single-row results. The repeated throughput 
regression observed during development did not recur in the final comparisons, 
but the latency increases above remain review considerations; these 
measurements do not establish a universal speedup.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed, including concurrent reads, writes and read/write 
interaction.
   - [x] added comments explaining accounting ownership, reservation ordering 
and snapshot lock ordering.
   - [x] added unit/component tests for lifecycle and failure paths.
   - [x] added integration tests and run them against a real 1C1D environment.
   
   ##### Key changed/added classes
   
   `WriteMemoryBlock`, `WriteMemoryManager`, `AbstractMemTable`, 
`TsFileProcessor`, `DataRegionInfo`, `TsFileProcessorInfo`, 
`MemTableFlushTask`, `SystemInfo`, `MemoryControlledWALEntryQueue`, 
`PrimitiveArrayManager`, `ReadOnlyMemoryBlock`, `MemoryManager`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to