Caideyipi opened a new pull request, #18344:
URL: https://github.com/apache/iotdb/pull/18344
## Description
### Problem
Pipe TsFile send paths accounted for the configured read buffer size before
reading from the file. As a result:
- The final partial piece was charged as a full buffer.
- The final EOF probe was charged as another full buffer.
- Files no larger than one read buffer were charged twice their actual size.
This reduced the effective global TsFile send rate below the configured
limit and inflated the PIPE_TSFILE_SEND_DISK_IO metric.
### Fix
Read each file piece first and account for the actual readLength only when
data is returned. The change covers:
- Common sync and air-gap file transfer paths.
- Data-region sync and async TsFile transfer paths.
- The batched data-region air-gap TsFile transfer path.
The async handler now uses a small overridable helper so the accounting
behavior can be tested without invoking the global limiter.
### Tests
mvn -o -nsu -Ddevelocity.off=true -DskipTests compile -pl
iotdb-core/node-commons,iotdb-core/datanode
mvn -o -nsu -Ddevelocity.off=true test -pl iotdb-core/datanode
-Dtest=IoTDBDataRegionAirGapSinkTest#testTransferTsFileBatchOverAirGap,PipeTransferTsFileHandlerRateLimitTest
Both targeted tests verify that the accounted bytes equal the actual
transferred/read bytes and that EOF is not accounted for.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- IoTDBSslSyncSink
- IoTDBAirGapSink
- IoTDBDataRegionSyncSink
- PipeTransferTsFileHandler
- IoTDBDataRegionAirGapSink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
