FrankChen021 commented on code in PR #20806:
URL: https://github.com/apache/druid/pull/20806#discussion_r4466295670
##########
processing/src/main/java/org/apache/druid/segment/loading/DeepStorageSegmentConfig.java:
##########
@@ -40,34 +45,62 @@
*/
public class DeepStorageSegmentConfig
{
- @JsonProperty
+ private static final Logger log = new Logger(DeepStorageSegmentConfig.class);
+
+ @JsonProperty("compressionFormat")
@Nullable
- private final Boolean zip;
+ private final CompressionUtils.Format compressionFormat;
@JsonCreator
- public DeepStorageSegmentConfig(@JsonProperty("zip") @Nullable Boolean zip)
+ public DeepStorageSegmentConfig(
+ @JsonProperty("zip") @Nullable Boolean zip,
+ @JsonProperty("compressionFormat") @Nullable CompressionUtils.Format
compressionFormat
+ )
{
- this.zip = zip;
+ if (zip != null) {
+ log.warn("druid.storage.zip is deprecated, use
druid.storage.compressionFormat instead.");
+ }
+ if (compressionFormat != null) {
+ this.compressionFormat = compressionFormat;
+ } else if (zip != null) {
+ this.compressionFormat = zip ? CompressionUtils.Format.ZIP :
CompressionUtils.Format.NONE;
+ } else {
+ this.compressionFormat = null;
+ }
}
public DeepStorageSegmentConfig()
{
- this(null);
+ this(null, null);
}
/**
- * Whether segments should be written as a zip file, falling back to {@code
defaultZip} when
- * {@code druid.storage.zip} has not been set.
+ * Resolves the effective compression format to use, falling back to {@code
defaultFormat} when neither
+ * {@code druid.storage.compressionFormat} nor {@code druid.storage.zip} has
been set.
+ * <p>
+ * When {@code supportedFormats} is non-empty, throws {@link IAE} if the
resolved format is not among them.
*/
- public boolean isZip(boolean defaultZip)
+ public CompressionUtils.Format getEffectiveCompressionFormat(
Review Comment:
[P2] Existing DeepStorageSegmentConfig APIs are removed
**Finding:** This replacement removes the public Boolean constructor,
isZip(boolean), and getZip() APIs that exist on the master base class.
Extensions or integrations compiled against the current API can fail to compile
or hit NoSuchMethodError after upgrading, even though the legacy storage
configuration is described as backward compatible.
**Suggestion:** Keep deprecated delegating constructors and accessors for
the existing Java API, or explicitly defer their removal to a breaking release.
##########
extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentPusher.java:
##########
@@ -181,52 +206,64 @@ private DataSegment pushToFilePath(File inDir,
DataSegment segment, String outIn
segment.getId(),
outIndexFilePath
);
- Path tmpIndexFile = new Path(StringUtils.format(
+ // For NONE, tmpPath is a directory; for ZIP/LZ4, tmpPath is a compressed
file.
+ final Path tmpPath = new Path(StringUtils.format(
"%s/%s/%s/%s_index%s",
fullyQualifiedStorageDirectory.get(),
segment.getDataSource(),
UUIDUtils.generateUuid(),
segment.getShardSpec().getPartitionNum(),
- config.getCompressionFormat().getSuffix()
+ compressionFormat == CompressionUtils.Format.NONE ? "" :
compressionFormat.getSuffix()
));
- FileSystem fs = tmpIndexFile.getFileSystem(hadoopConfig);
+ final FileSystem fs = tmpPath.getFileSystem(hadoopConfig);
- fs.mkdirs(tmpIndexFile.getParent());
- log.debug("Compressing files from[%s] to [%s]", inDir, tmpIndexFile);
+ fs.mkdirs(tmpPath.getParent());
Review Comment:
[P2] Temp parent is created outside the cleanup scope
**Finding:** The temporary UUID parent is created before the try/finally
that removes it. If HDFS reports an exception after creating that parent, the
finally block is never entered; the retry loop then uses another UUID and can
leave the failed attempt's directory behind, leaking HDFS storage on publish
failures.
**Suggestion:** Move temporary-parent creation into the protected staging
scope so every failed attempt, including mkdir failures, removes its UUID
directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
