luoluoyuyu opened a new pull request, #19324:
URL: https://github.com/apache/iotdb/pull/19324

   ## Description
   
   `AlignedWritableMemChunk#encodeWorkingAlignedTVList()` used two passes for 
duplicate timestamps:
   the first pass built chunk/page ranges and filled a `boolean[rowCount]` 
marking every duplicate
   row, then the encoder traversed the same rows again per column and once more 
to write the time
   page.
   
   This PR keeps the page/chunk range construction but removes the per-row 
duplicate bitmap. The
   encoder now makes a single row-major pass over each page, keeps a pending 
timestamp group, tracks
   the last non-null value of each column, and writes the group when its 
timestamp ends. A group that
   crosses a page or chunk boundary is carried into the next page/chunk, so 
value pages and time pages
   stay aligned. The dense path (no duplicate timestamps, no deleted/all-null 
rows, full schema) still
   uses the segment-batch encoder.
   
   ### Example
   
   Sorted input with page cap 2 and chunk cap 3:
   
   | time | s0 | s1 |
   | ---: | ---: | ---: |
   | 1 | 1 | null |
   | 1 | null | 11 |
   | 1 | null | null |
   | 2 | 2 | 22 |
   | 3 | 3 | null |
   | 3 | null | 33 |
   | 5 | 5 | null |
   | 5 | null | 55 |
   | 5 | null | null |
   | 5 | null | null |
   | 6 | 6 | 66 |
   
   Timestamp 5 has four duplicate rows, so its group spans page and chunk 
boundaries. Both before and
   after encode the same output:
   
   | time | s0 | s1 |
   | ---: | ---: | ---: |
   | 1 | 1 | 11 |
   | 2 | 2 | 22 |
   | 3 | 3 | 33 |
   | 5 | 5 | 55 |
   | 6 | 6 | 66 |
   
   Before, producing this required a duplicate-detection scan, then a 
value-column scan, then a time
   scan. After, the duplicate merge happens while the row-major encoding pass 
walks the page once.
   
   ### Verification
   
   - `MemTableFlushTaskTest`: 19 tests, including new cases for duplicate 
groups crossing page/chunk
     boundaries, with and without measurement deletion and `ignoreAllNullRows`.
   - `AlignedWritableMemChunkTest`: 6 tests.
   - `AlignedTVListTest`: 29 tests.
   - Opt-in `AlignedFlushDuplicateTimeCheckPerformanceTest` smoke run: legacy 
and single-pass results
     are identical; the sampled optimized/legacy CPU ratio was about 38% (smoke 
only, not a benchmark
     claim).
   - `mvn spotless:check -pl iotdb-core/datanode`: passed.
   - `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale`: passed.
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the intent where the duplicate-group carry 
logic is non-obvious.
   - [x] added or updated unit tests to cover the changed code paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to