luoluoyuyu opened a new pull request, #19059:
URL: https://github.com/apache/iotdb/pull/19059

   ## Description
   
   ### Problem
   
   `PipeTsFileEpochProgressIndexKeeper` keeps pipe tsFile dedup state per data 
region (`regionId -> taskScope -> tsFilePath -> TsFileResource`). Entries are 
only removed by pipe event callbacks and by the realtime source's `close()`. 
`StorageEngine.deleteDataRegion()` removes the region, its files, WAL, 
snapshots and the receiver-side resource, but never removes this source-side 
dedup state. After a region is deleted (`DROP DATABASE`, region migration 
cleanup), its entries stay in the DataNode heap for the rest of the process 
lifetime: memory grows on every create/drop cycle, and a re-created region can 
be matched against stale progress indexes.
   
   ### Solution
   
   - Add `PipeTsFileEpochProgressIndexKeeper.clearProgressIndex(int 
dataRegionId)`, which drops all task scopes of one region at once.
   - Call it from `StorageEngine.deleteDataRegion()` right after 
`region.markDeleted()`, i.e. before the region files are removed, so the 
cleanup also happens when the rest of the deletion fails. The cleanup is logged 
through a new `StorageEngineMessages` constant (en and zh).
   
   ### Tests
   
   - UT 
`PipeTsFileEpochProgressIndexKeeperTest#testClearProgressIndexRemovesAllTaskScopesOfDataRegion`:
 entries of two different task scopes of the same region are all removed.
   - UT `DataRegionTest#testDeletedRegionClearsItsPipeProgressIndexes`: 
registers an index, runs the real `StorageEngine.deleteDataRegion()`, asserts 
the entry is gone.
   - IT `IoTDBPipeProgressIndexCleanupIT` (`LocalStandaloneIT`, 1C1D, 
`SimpleIT`): creates a database, writes and flushes data, drops the database, 
and waits for the DataNode log to report the cleanup. It fails on the pre-fix 
build (timeout, no cleanup record) and passes with the fix (`StorageEngine:887 
- Cleared pipe progress indexes of the deleted data region 1`).
   
   Verified locally: keeper UT 5/5, `DataRegionTest` method 1/1, IT 1/1, 
`spotless:check` for `iotdb-core/datanode` and `integration-test`, 
`test-compile` for the default locale and for `-P with-zh-locale`.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] added integration tests.
   - [x] been tested in a test IoTDB cluster.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - 
`org.apache.iotdb.db.pipe.source.dataregion.realtime.assigner.PipeTsFileEpochProgressIndexKeeper`
   - `org.apache.iotdb.db.storageengine.StorageEngine`
   - `org.apache.iotdb.db.i18n.StorageEngineMessages` (en / zh)
   - 
`org.apache.iotdb.db.pipe.source.dataregion.realtime.assigner.PipeTsFileEpochProgressIndexKeeperTest`
   - `org.apache.iotdb.db.storageengine.dataregion.DataRegionTest`
   - `org.apache.iotdb.db.it.IoTDBPipeProgressIndexCleanupIT`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to