luoluoyuyu opened a new pull request, #19324:
URL: https://github.com/apache/iotdb/pull/19324
## Description
`AlignedWritableMemChunk#encodeWorkingAlignedTVList()` used two passes for
duplicate timestamps:
the first pass built chunk/page ranges and filled a `boolean[rowCount]`
marking every duplicate
row, then the encoder traversed the same rows again per column and once more
to write the time
page.
This PR keeps the page/chunk range construction but removes the per-row
duplicate bitmap. The
encoder now makes a single row-major pass over each page, keeps a pending
timestamp group, tracks
the last non-null value of each column, and writes the group when its
timestamp ends. A group that
crosses a page or chunk boundary is carried into the next page/chunk, so
value pages and time pages
stay aligned. The dense path (no duplicate timestamps, no deleted/all-null
rows, full schema) still
uses the segment-batch encoder.
### Example
Sorted input with page cap 2 and chunk cap 3:
| time | s0 | s1 |
| ---: | ---: | ---: |
| 1 | 1 | null |
| 1 | null | 11 |
| 1 | null | null |
| 2 | 2 | 22 |
| 3 | 3 | null |
| 3 | null | 33 |
| 5 | 5 | null |
| 5 | null | 55 |
| 5 | null | null |
| 5 | null | null |
| 6 | 6 | 66 |
Timestamp 5 has four duplicate rows, so its group spans page and chunk
boundaries. Both before and
after encode the same output:
| time | s0 | s1 |
| ---: | ---: | ---: |
| 1 | 1 | 11 |
| 2 | 2 | 22 |
| 3 | 3 | 33 |
| 5 | 5 | 55 |
| 6 | 6 | 66 |
Before, producing this required a duplicate-detection scan, then a
value-column scan, then a time
scan. After, the duplicate merge happens while the row-major encoding pass
walks the page once.
### Verification
- `MemTableFlushTaskTest`: 19 tests, including new cases for duplicate
groups crossing page/chunk
boundaries, with and without measurement deletion and `ignoreAllNullRows`.
- `AlignedWritableMemChunkTest`: 6 tests.
- `AlignedTVListTest`: 29 tests.
- Opt-in `AlignedFlushDuplicateTimeCheckPerformanceTest` smoke run: legacy
and single-pass results
are identical; the sampled optimized/legacy CPU ratio was about 38% (smoke
only, not a benchmark
claim).
- `mvn spotless:check -pl iotdb-core/datanode`: passed.
- `mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale`: passed.
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the intent where the duplicate-group carry
logic is non-obvious.
- [x] added or updated unit tests to cover the changed code paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]