Caideyipi opened a new pull request, #18724:
URL: https://github.com/apache/iotdb/pull/18724
## Description
### Reduce duplicate timestamp flush overhead
- Cache the unsorted value index once per row instead of resolving it
repeatedly.
- Reuse the null check already performed while selecting the last non-null
value for a duplicate timestamp.
- Replace per-column Pair<Long, Integer> state with primitive arrays,
avoiding boxing in the hot loop.
- Apply the same logic to both the normal aligned flush path and the
deleted-measurement mapping path.
The duplicate timestamp merge behavior is unchanged: each value column keeps
the last non-null value at a timestamp.
### Tests
- Added correctness coverage for duplicate timestamps with the full
measurement set.
- Added correctness coverage after deleting a measurement, exercising column
remapping.
- Added an opt-in manual performance UT. It stays skipped during normal test
runs and can be enabled with:
mvn test -pl iotdb-core/datanode
-Dtest=AlignedFlushDuplicateTimeCheckPerformanceTest
-Diotdb.aligned.flush.duplicate-check.perf.enabled=true
Benchmark workload: 16,384 rows, 8 aligned columns, four rows per duplicated
timestamp, 128 scans per round, five rounds.
| Implementation | CPU per scan | Allocated per scan |
| --- | ---: | ---: |
| Legacy | 1.099 ms | 3,846,576 bytes |
| Optimized | 0.366 ms | 128 bytes |
The optimized loop used about 33.33% of the legacy CPU time in this targeted
benchmark.
Verification:
mvn -nsu test -pl iotdb-core/datanode
'-Dtest=MemTableFlushTaskTest,AlignedFlushDuplicateTimeCheckPerformanceTest'
13 tests ran: 12 passed and the opt-in performance UT was skipped as
expected.
---
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the intent where the optimized duplicate
merge is non-obvious.
- [x] added or updated unit tests to cover the changed code paths.
---
##### Key changed/added classes in this PR
- AlignedWritableMemChunk
- MemTableFlushTaskTest
- AlignedFlushDuplicateTimeCheckPerformanceTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]