luoluoyuyu opened a new pull request, #19319:
URL: https://github.com/apache/iotdb/pull/19319
## Description
`AlignedTVList#getAvgPointSizeOfLargestColumn()` is used during every flush
to derive the chunk
cap for the widest binary column. It previously called
`getColumnValueCnt()`, which walked all
`rowCount` rows and called `isNullValue()` for each row of that column. On a
large memtable this
adds an `O(rowCount)` scan to the flush path for every
`AlignedWritableMemChunk`.
This PR keeps the non-null value count of each column incrementally, so the
flush path reads the
count in `O(1)`.
### Example
Assume a memtable `AlignedTVList` with 1,000,000 rows, where the largest
binary column contains
800,000 non-null values and 200,000 nulls:
- Before: each flush of that memtable scanned 1,000,000 rows with
`isNullValue()` to obtain
`columnValueCnt = 800,000`.
- After: writes/deletes maintain `nonNullValueCounts[column]`, so the same
flush reads
`columnValueCnt = 800,000` in `O(1)` and computes the average point size
from
`memoryBinaryChunkSize / columnValueCnt` as before.
### Implementation
- Added `int[] nonNullValueCounts` to `AlignedTVList` and kept it in sync in:
- `putAlignedValue` (single-row writes),
- `putAlignedValues` / `markNullBitmapRange` (batch writes, including
input null bitmaps and
failed `TSStatus` rows),
- `delete` and `deleteColumn`,
- `clone`, `cloneForFlushSort` and the partial-clone transfer,
- `releaseNonQueryColumns`, `clear` and `extendColumns`.
- `getColumnValueCnt()` now returns the maintained counter instead of
scanning the column.
- The counter array is included in `calculateContainerRamCost()` so memory
accounting stays
complete.
- Added regression tests for single/batch writes with nulls and failed
statuses, range deletion,
whole-column deletion, clone/partial clone, WAL deserialization and column
release.
### Verification
- `AlignedTVListTest`: 32 tests, 0 failures.
- `AlignedWritableMemChunkTest`, `MemTableMemoryBlockTest`,
`FragmentInstanceExecutionTest`: 14 tests, 0 failures.
- `mvn spotless:check -pl iotdb-core/datanode`: passed.
- `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale`: passed.
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code.
- [x] added unit tests or modified existing tests to cover new code paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]