Caideyipi opened a new pull request, #18330:
URL: https://github.com/apache/iotdb/pull/18330
## Description
### Problem
#18249 made aligned MemTable bitmaps lazy, but the write-memory estimator
still defensively reserved a bitmap object and bitmap-list reference for every
value block. That reservation can make memory accounting larger than the
objects that are actually materialized and may reject writes unnecessarily.
A newly extended column also keeps its historical value arrays as `null`.
Therefore, this accounting model should not charge the additional 4-byte object
reference for each historical null placeholder; the reference is charged only
when the value array is materialized.
### Fix
- Remove the unconditional per-block bitmap reservation and account for
bitmap references/objects only when they are actually created.
- Track materialized value-array and bitmap costs incrementally in
`AlignedTVList`, with an O(1) `getRamSize()` for the write path.
- Reconcile aligned TVList RAM after row, rows, and tablet writes, including
mixed-alignment rows, multi-device tablets, and sorted TVLists.
The correction is:
```text
actual aligned TVList RAM delta - estimated aligned MemTable increment
```
A positive correction is reserved in both `DataRegionInfo` and the MemTable;
a negative correction is released from both and refreshes the storage-group
status. Chunk metadata and variable-length Binary payload accounting are
intentionally unchanged.
### Performance
The new performance UT is disabled by default and can be enabled with:
```shell
mvn -pl iotdb-core/datanode test \
-Dtest=AlignedBitmapMemoryAccountingPerformanceTest \
-Diotdb.aligned.bitmap.accounting.perf.enabled=true
```
Default benchmark setup: 64 columns, 256 rows per batch, 200 warmups, 2,000
batches per round, and 5 rounds. Reconciliation is repeated internally to avoid
Windows CPU timer granularity and then normalized per batch.
| Scenario | Reconciliation CPU | Write CPU | CPU ratio | Reconciliation
allocation |
| --- | ---: | ---: | ---: | ---: |
| Dense | 0.038 us/batch | 23.438 us/batch | 0.16% | 0 B/batch |
| Null-heavy | 0.031 us/batch | 39.063 us/batch | 0.08% | 0 B/batch |
The measured reconciliation overhead is therefore negligible for both dense
and null-heavy writes.
### Verification
- `AlignedTVListTest`: 15 passed
- `TsFileProcessorTest`: 20 passed
- Performance UT default-off behavior: 1 skipped as expected
- Performance UT enabled: 1 passed
- `IoTDBAlignedTVListLazyAllocationIT`: 1 passed on a packaged 1C1D cluster
- Datanode and integration-test Checkstyle: 0 violations
- Spotless and `git diff --check`: passed
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the intent of the accounting and
reconciliation logic.
- [x] added or updated unit tests to cover the new code paths.
- [x] added/updated an integration test.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes
- `AlignedTVList`: actual materialized value-array/bitmap RAM tracking and
constant-time RAM lookup.
- `TsFileProcessor`: post-write aligned RAM reconciliation.
- `AlignedBitmapMemoryAccountingPerformanceTest`: opt-in reconciliation
overhead benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
