hongzhi-gao opened a new pull request, #18848:
URL: https://github.com/apache/iotdb/pull/18848
## Description
Write-memory accounting is spread across MemTables, TsFileProcessors and
DataRegions, requiring callers to update several counters together. This change
introduces named, categorized owner blocks that propagate usage to their
accounting scope and tie rollback/release to the owner's lifecycle.
### Accounting and lifecycle
- Add `WriteMemoryBlock` and `WriteMemoryManager` for MemTable arrays,
Binary values, recovered totals, TsFile metadata and flush temporary
reservations. Pending reservations count toward admission but become used
MemTable memory only after admission succeeds. WAL failures roll back the
admitted deltas.
- Replace positional write-memory deltas with named fields across row,
mixed-row and split-tablet paths. MemTable release closes its blocks and
updates the region total once; query-held TVLists retain their existing
resource ownership.
- Preserve the WAL snapshot's aggregate memory field. Recovery places that
aggregate in a separate category rather than guessing an array/Binary breakdown.
- Scope flush reservations to actual worker lifetime: propagate worker
failures, cancel and await both workers, clear temporary references, then
release the reservation. Budget reload retains outstanding reservations.
- Store each WAL queue entry's original charge and refund it on dequeue or
interrupted enqueue.
### Array-pool accounting and hot paths
Count idle arrays by actual shallow size under the existing per-type queue
locks. A new `ReadOnlyMemoryBlock` exposes an exact snapshot to the existing
memory-manager tree; readers acquire queue locks in a fixed order. This avoids
an additional global atomic update on each array borrow/return.
`WriteMemoryBlock` uses a monitor-guarded `long`; region totals remain atomic.
Zero-delta writes skip reservation creation while retaining pressure checks.
SQL, configured memory limits, file/WAL formats and the existing admission
policy are unchanged. Accounting follows write ownership, not total JVM heap
liveness. New exception messages are provided in both English and Chinese.
### Verification
Executed remotely on Linux with JDK 21 (production compilation targets Java
17), against the exact submitted source:
- Spotless and focused Maven tests: **122 passed**, zero
failures/errors/skips (7 Node Commons, 115 DataNode).
- Full-reactor `mvn test-compile -DskipTests` and `mvn test-compile
-DskipTests -P with-zh-locale`: passed. Packaging prerequisites were built with
`mvn install -DskipTests -pl
iotdb-core/datanode,iotdb-core/confignode,iotdb-client/cli -am`.
- **4 lifecycle ITs passed**, using the repository's real 1C1D fixtures
through JUnitCore, rather than Maven Failsafe. Coverage includes 63/64/65 array
boundaries, sparse/new aligned columns, repeated flush/unsequence overwrite,
oversized-WAL row/tablet rejection and retry, forced DataNode restart/recovery,
and table writes split across devices/time partitions. Tests verify query
results and live database write-memory metrics.
- Targeted tests also cover concurrent reservations/snapshots, query-held
TVList ownership, interrupted WAL enqueue, and flush failure/interruption
cleanup.
The focused Maven test selection was:
```sh
mvn test -pl iotdb-core/node-commons,iotdb-core/datanode \
-Dtest=MemoryBlockTest,MemoryManagerTest,ReadOnlyMemoryBlockTest,WriteMemoryBlockTest,WriteMemoryLifecycleTest,MemTableMemoryBlockTest,TsFileProcessorTest,MemTableFlushTaskTest,PrimitiveArrayManagerTest,BooleanTVListTest,IntTVListTest,LongTVListTest,FloatTVListTest,DoubleTVListTest,BinaryTVListTest,AlignedTVListTest
\
-Dsurefire.failIfNoSpecifiedTests=false
```
Initial environment failures were resolved before reporting these results:
missing distribution ZIP prerequisites, and JMX port collisions during IT node
startup. The unchanged ITs passed after rerunning with free ports.
### Performance observations
Compared with `eb0e3f1f2080e34bf5593bfd1cf63b62ecdba2e0`, using the same
workload/configuration and three interleaved runs per version:
| Workload (4 writers) | Median throughput | P50 RPC latency | P99 RPC
latency |
| --- | ---: | ---: | ---: |
| Non-aligned Tablet, 1024 rows | -0.24% | +3.88% | +0.10% |
| Aligned single row | +0.50% | -0.73% | +3.19% |
| Aligned Tablet, 1024 rows | +4.36% | -4.60% | -3.58% |
Environment: shared Xeon Gold 6130 KVM host, 32 vCPUs, JDK 21,
1C1D/SimpleConsensus, ASYNC WAL, memory control enabled, 2 GiB DataNode heap,
array size 64, MemTable threshold 32 MiB, compaction and timed flush disabled.
Fixed per-writer warmup/measurement calls were 50k/100k for non-aligned Tablet,
125k/250k for aligned rows, and 20k/40k for aligned Tablet. Each run checked
acknowledged row counts through queries after a final FLUSH.
These are controlled class-overlay comparisons, not two independent clean
distribution builds. The throughput window excludes the final FLUSH; it does
not measure synchronous durability throughput. Shared-host variability and
existing QueryId waits affect single-row results. The repeated throughput
regression observed during development did not recur in the final comparisons,
but the latency increases above remain review considerations; these
measurements do not establish a universal speedup.
<hr>
This PR has:
- [x] been self-reviewed, including concurrent reads, writes and read/write
interaction.
- [x] added comments explaining accounting ownership, reservation ordering
and snapshot lock ordering.
- [x] added unit/component tests for lifecycle and failure paths.
- [x] added integration tests and run them against a real 1C1D environment.
##### Key changed/added classes
`WriteMemoryBlock`, `WriteMemoryManager`, `AbstractMemTable`,
`TsFileProcessor`, `DataRegionInfo`, `TsFileProcessorInfo`,
`MemTableFlushTask`, `SystemInfo`, `MemoryControlledWALEntryQueue`,
`PrimitiveArrayManager`, `ReadOnlyMemoryBlock`, `MemoryManager`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]