jackjlli commented on a change in pull request #5764:
URL: https://github.com/apache/incubator-pinot/pull/5764#discussion_r462575993
##########
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentFetcherAndLoader.java
##########
@@ -184,39 +186,48 @@ private boolean isNewSegmentMetadata(String
tableNameWithType, OfflineSegmentZKM
private String downloadSegmentToLocal(String uri, PinotCrypter crypter,
String tableName, String segmentName)
throws Exception {
- File tempDir = new File(new
File(_instanceDataManager.getSegmentFileDirectory(), tableName),
- "tmp-" + segmentName + "-" + UUID.randomUUID());
- FileUtils.forceMkdir(tempDir);
- File tempDownloadFile = new File(tempDir, segmentName + ENCODED_SUFFIX);
- File tempTarFile = new File(tempDir, segmentName + TAR_GZ_SUFFIX);
- File tempSegmentDir = new File(tempDir, segmentName);
- try {
- SegmentFetcherFactory.fetchSegmentToLocal(uri, tempDownloadFile);
- if (crypter != null) {
- crypter.decrypt(tempDownloadFile, tempTarFile);
- } else {
- tempTarFile = tempDownloadFile;
- }
-
- LOGGER
- .info("Downloaded tarred segment: {} for table: {} from: {} to: {},
file length: {}", segmentName, tableName,
- uri, tempTarFile, tempTarFile.length());
+ // Even if the tar file has been downloaded successfully, the file itself
could be corrupted during the transmission.
Review comment:
I just filed another PR to add the server metric:
https://github.com/apache/incubator-pinot/pull/5768.
We can hold off this PR a bit and see how often the exception happens on
untarring segments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
