luoluoyuyu opened a new pull request, #19059: URL: https://github.com/apache/iotdb/pull/19059
## Description ### Problem `PipeTsFileEpochProgressIndexKeeper` keeps pipe tsFile dedup state per data region (`regionId -> taskScope -> tsFilePath -> TsFileResource`). Entries are only removed by pipe event callbacks and by the realtime source's `close()`. `StorageEngine.deleteDataRegion()` removes the region, its files, WAL, snapshots and the receiver-side resource, but never removes this source-side dedup state. After a region is deleted (`DROP DATABASE`, region migration cleanup), its entries stay in the DataNode heap for the rest of the process lifetime: memory grows on every create/drop cycle, and a re-created region can be matched against stale progress indexes. ### Solution - Add `PipeTsFileEpochProgressIndexKeeper.clearProgressIndex(int dataRegionId)`, which drops all task scopes of one region at once. - Call it from `StorageEngine.deleteDataRegion()` right after `region.markDeleted()`, i.e. before the region files are removed, so the cleanup also happens when the rest of the deletion fails. The cleanup is logged through a new `StorageEngineMessages` constant (en and zh). ### Tests - UT `PipeTsFileEpochProgressIndexKeeperTest#testClearProgressIndexRemovesAllTaskScopesOfDataRegion`: entries of two different task scopes of the same region are all removed. - UT `DataRegionTest#testDeletedRegionClearsItsPipeProgressIndexes`: registers an index, runs the real `StorageEngine.deleteDataRegion()`, asserts the entry is gone. - IT `IoTDBPipeProgressIndexCleanupIT` (`LocalStandaloneIT`, 1C1D, `SimpleIT`): creates a database, writes and flushes data, drops the database, and waits for the DataNode log to report the cleanup. It fails on the pre-fix build (timeout, no cleanup record) and passes with the fix (`StorageEngine:887 - Cleared pipe progress indexes of the deleted data region 1`). Verified locally: keeper UT 5/5, `DataRegionTest` method 1/1, IT 1/1, `spotless:check` for `iotdb-core/datanode` and `integration-test`, `test-compile` for the default locale and for `-P with-zh-locale`. <hr> This PR has: - [x] been self-reviewed. - [x] added unit tests or modified existing tests to cover new code paths. - [x] added integration tests. - [x] been tested in a test IoTDB cluster. <hr> ##### Key changed/added classes (or packages if there are too many classes) in this PR - `org.apache.iotdb.db.pipe.source.dataregion.realtime.assigner.PipeTsFileEpochProgressIndexKeeper` - `org.apache.iotdb.db.storageengine.StorageEngine` - `org.apache.iotdb.db.i18n.StorageEngineMessages` (en / zh) - `org.apache.iotdb.db.pipe.source.dataregion.realtime.assigner.PipeTsFileEpochProgressIndexKeeperTest` - `org.apache.iotdb.db.storageengine.dataregion.DataRegionTest` - `org.apache.iotdb.db.it.IoTDBPipeProgressIndexCleanupIT` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
