FrankChen021 commented on code in PR #20806:
URL: https://github.com/apache/druid/pull/20806#discussion_r4434062725
##########
extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:
##########
@@ -74,52 +86,59 @@ public void kill(DataSegment segment) throws
SegmentLoadingException
try (final FileSystem fs = segmentPath.getFileSystem(config)) {
final String filename = segmentPath.getName();
final CompressionUtils.Format compressionFormat =
CompressionUtils.Format.fromFileName(filename);
- if (compressionFormat != CompressionUtils.Format.ZIP &&
compressionFormat != CompressionUtils.Format.LZ4) {
- throw new SegmentLoadingException("Unknown file type[%s]",
segmentPath);
- } else {
+ final boolean isUncompressed = compressionFormat == null &&
isUncompressedDirectorySegment(filename);
Review Comment:
[P2] Treat extension-bearing NONE paths as directories
**Finding:** When compressionFormat=none and pushToPath receives a suffix
ending in index.zip or index.lz4, the existing shortcut in
HdfsDataSegmentPusher uses that suffix as the final path, while
pushToFilePathNone writes a directory there. Because this line lets the
filename-derived format win, isUncompressed is false and kill calls
fs.delete(segmentPath, false), which cannot remove the non-empty directory.
storageDirSuffix is caller-provided and pushToPath explicitly supports
suffixes, so this can leave a published NONE segment undeletable.
**Suggestion:** Make the complete-path shortcut conditional on the selected
compression format, or classify directory paths from FileStatus before
inferring a format from the filename; add a regression that pushes NONE with an
extension-bearing suffix and then kills it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
