FrankChen021 commented on code in PR #18982:
URL: https://github.com/apache/druid/pull/18982#discussion_r2797196403
##########
extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPuller.java:
##########
@@ -235,27 +248,16 @@ FileUtils.FileCopyResult getSegmentFiles(final Path path,
final File outDir) thr
catch (Exception e) {
throw new RuntimeException(e);
}
- } else if (CompressionUtils.isZip(path.getName())) {
+ }
- // -------- zip ---------
+ // Try to detect format from file extension and decompress
+ final CompressionUtils.Format format =
CompressionUtils.Format.fromFileName(path.getName());
+ if ((format == CompressionUtils.Format.ZIP || format ==
CompressionUtils.Format.LZ4)) {
+ long startTime = System.currentTimeMillis();
- final FileUtils.FileCopyResult result = CompressionUtils.unzip(
- new ByteSource()
- {
- @Override
- public InputStream openStream() throws IOException
- {
- return getInputStream(path);
- }
- }, outDir, shouldRetryPredicate(), false
- );
+ final FileUtils.FileCopyResult result =
format.decompressDirectory(getInputStream(path), outDir);
- log.info(
- "Unzipped %d bytes from [%s] to [%s]",
- result.size(),
- path.toString(),
- outDir.getAbsolutePath()
- );
+ emitMetrics(format, result.size(), System.currentTimeMillis() -
startTime);
Review Comment:
currently only zip is saved in hdfs, gz and dir should be old code path. no
need to support it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
