hongzhi-gao opened a new pull request, #18019: URL: https://github.com/apache/iotdb/pull/18019
# Optimize Snapshot Transfer Performance and Logging ## Description ### 1. Buffer Reuse on the Snapshot Send Path - `SnapshotFragmentReader` previously allocated a fresh 10MB `ByteBuffer` per file. - A snapshot may contain hundreds of thousands of tiny files, leading to significant GC pressure and high allocation costs. - **Improvement**: The buffer is now caller-supplied and reused across all files (safe because each fragment is serialized synchronously before the next read clears the buffer). ### 2. Lower Per-File Logging Cost - The full file-name list is now built and emitted only at `debug` level. - Per-file progress logging is throttled to at most once per configured interval, instead of emitting one `INFO` line per file. ### 3. Make the Progress-Log Interval Configurable - Added `data_region_iot_snapshot_transmission_progress_log_interval_ms` (default: `5000`, `<= 0` logs every file, hot-reloadable). - Plumbed through `IoTConsensusConfig.Replication`. --- ## PR Checklist - [x] Been self-reviewed - [x] Added comments explaining the "why" and intent of the code wherever it would not be obvious to an unfamiliar reader - [x] Added unit tests or modified existing tests to cover new code paths --- ## Key Changed/Added Classes (or Packages) - `SnapshotFragmentReader` / `IoTConsensusServerImpl` (`consensus.iot.snapshot`) - `IoTConsensusConfig.Replication` - `IoTDBConfig` / `IoTDBDescriptor` / `DataRegionConsensusImpl` - `SnapshotFragmentReaderTest` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
