luoluoyuyu opened a new pull request, #19319:
URL: https://github.com/apache/iotdb/pull/19319

   ## Description
   
   `AlignedTVList#getAvgPointSizeOfLargestColumn()` is used during every flush 
to derive the chunk
   cap for the widest binary column. It previously called 
`getColumnValueCnt()`, which walked all
   `rowCount` rows and called `isNullValue()` for each row of that column. On a 
large memtable this
   adds an `O(rowCount)` scan to the flush path for every 
`AlignedWritableMemChunk`.
   
   This PR keeps the non-null value count of each column incrementally, so the 
flush path reads the
   count in `O(1)`.
   
   ### Example
   
   Assume a memtable `AlignedTVList` with 1,000,000 rows, where the largest 
binary column contains
   800,000 non-null values and 200,000 nulls:
   
   - Before: each flush of that memtable scanned 1,000,000 rows with 
`isNullValue()` to obtain
     `columnValueCnt = 800,000`.
   - After: writes/deletes maintain `nonNullValueCounts[column]`, so the same 
flush reads
     `columnValueCnt = 800,000` in `O(1)` and computes the average point size 
from
     `memoryBinaryChunkSize / columnValueCnt` as before.
   
   ### Implementation
   
   - Added `int[] nonNullValueCounts` to `AlignedTVList` and kept it in sync in:
     - `putAlignedValue` (single-row writes),
     - `putAlignedValues` / `markNullBitmapRange` (batch writes, including 
input null bitmaps and
       failed `TSStatus` rows),
     - `delete` and `deleteColumn`,
     - `clone`, `cloneForFlushSort` and the partial-clone transfer,
     - `releaseNonQueryColumns`, `clear` and `extendColumns`.
   - `getColumnValueCnt()` now returns the maintained counter instead of 
scanning the column.
   - The counter array is included in `calculateContainerRamCost()` so memory 
accounting stays
     complete.
   - Added regression tests for single/batch writes with nulls and failed 
statuses, range deletion,
     whole-column deletion, clone/partial clone, WAL deserialization and column 
release.
   
   ### Verification
   
   - `AlignedTVListTest`: 32 tests, 0 failures.
   - `AlignedWritableMemChunkTest`, `MemTableMemoryBlockTest`,
     `FragmentInstanceExecutionTest`: 14 tests, 0 failures.
   - `mvn spotless:check -pl iotdb-core/datanode`: passed.
   - `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale`: passed.
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code.
   - [x] added unit tests or modified existing tests to cover new code paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to