VGalaxies opened a new pull request, #19339:
URL: https://github.com/apache/iotdb/pull/19339

   ## Problem
   
   When an `AlignedTVList` receives out-of-order writes and is subsequently 
queried before flushing, query execution calls `AlignedTVList.sort()`, which 
allocates `indices` and `segmentMovedMap` without charging the owning 
`IMemTable`'s `structureMemoryBlock`.
   
   On a subsequent out-of-order write to the same working `AlignedTVList`:
   1. `TsFileProcessor` previously measured live `getStructureRamSize()` 
**before** the write (which included the unaccounted `segmentMovedMap` bytes 
allocated by query `sort()`).
   2. The out-of-order write cleared `segmentMovedMap`, causing post-write 
`getStructureRamSize()` to drop below the pre-write snapshot.
   3. Reconciling the negative delta against `structureMemoryBlock` released 
bytes that had never been charged, triggering a `WriteMemoryBlock` underflow 
(`IllegalArgumentException` / failed flush or insert).
   
   The same gap also applied when query-side clone-on-write replaced the 
working TVList via `AlignedTVList.clone()` or 
`AlignedTVList.commitPartialClone(PartialClonePlan)` without preserving the 
accounted RAM baseline on the replacement working TVList.
   
   ## Fix
   
   - Track the last array and structure RAM sizes reconciled into the owning 
`IMemTable`'s `WriteMemoryBlock`s (`ramSizeAccounted`, `accountedArrayRamSize`, 
`accountedStructureRamSize`) on `AlignedTVList`.
   - Initialize/reset this baseline on WAL deserialization 
(`AlignedTVList.deserialize`) and `clear()`, and preserve it across full and 
partial clone replacements (`clone()` and `commitPartialClone(...)`).
   - Update `TsFileProcessor` RAM reconciliation for aligned TVLists to diff 
post-write actual RAM against the previously accounted baseline rather than a 
live pre-write snapshot that may include query-allocated structures.
   
   ## Tests
   
   - `AlignedTVListTest`: added 
`testAccountedRamSizeLifecycleOnDeserializeAndClear` and 
`testAccountedRamSizePreservedAcrossCloneAndPartialClone` (31/31 pass).
   - `TsFileProcessorTest`: added 
`alignedStructureMemoryAccountingSurvivesRepeatedSortAndOutOfOrderWrites`, 
`alignedStructureMemoryAccountingSurvivesMultiDeviceSortAndOutOfOrderBatch`, 
and `alignedStructureMemoryAccountingSurvivesCloneAndPartialCloneReplacement` 
(33/33 pass).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to