somiljain2006 opened a new pull request, #20806: URL: https://github.com/apache/druid/pull/20806
Fixes #20767 ### Description This PR introduces a configurable compression format for deep storage segments, replacing the backend-specific boolean `druid.storage.zip` configuration with the more flexible `druid.storage.compressionFormat` setting. The change supports ZIP, LZ4 (where supported by the storage backend), and NONE (uncompressed directory-based storage), while retaining backward compatibility with the existing `druid.storage.zip` configuration. #### Centralized deep-storage compression configuration - Added `druid.storage.compressionFormat` to `DeepStorageSegmentConfig`. - Supported values are `zip`, `lz4`, and `none` (case-insensitive). - Retained `druid.storage.zip` as a deprecated compatibility property. - Preserved tri-state configuration semantics so that each backend can retain its existing default. - Added validation for conflicting legacy and new configuration values. - Deprecated `isZip(boolean)` while retaining it for Java API compatibility. When both properties are specified, compatible values are accepted, while conflicting configurations fail validation. #### Extended compression format support - Added `NONE` to `CompressionUtils.Format` to represent uncompressed directory-based storage. - Kept `NONE` out of the file-extension lookup so that filename-based detection continues to identify only actual compressed formats. - Prevented archive compression and decompression methods from being invoked with `NONE`. #### Migrated deep-storage pushers Migrated the Local, S3, Google Cloud Storage, and Azure segment pushers to use the centralized compression configuration. The existing backend defaults are preserved: - Local storage defaults to `NONE`. - S3, Google Cloud Storage, and Azure default to `ZIP`. Local storage and cloud storage pushers support `ZIP` and `NONE`. Unsupported formats are rejected during pusher initialization rather than failing later during segment publication. Existing ZIP publishing behavior, loadSpec generation, and stale-object cleanup behavior are retained. #### Added uncompressed directory support for HDFS - Moved the compression format configuration from `HdfsDataSegmentPusherConfig` into the shared `DeepStorageSegmentConfig`. - Preserved `ZIP` as the HDFS default. - Added support for `ZIP`, `LZ4`, and `NONE`. - For `NONE`, segments are staged in a temporary HDFS directory and published by renaming the completed directory into its final location. - The generated loadSpec points to the uncompressed segment directory. - Added cleanup of temporary directories when publishing fails. - Extended `HdfsDataSegmentKiller` to handle uncompressed directory-based segments while preserving path traversal validation. - Verified and tested the existing directory-loading support in `HdfsDataSegmentPuller`. #### Documentation Updated the deep-storage configuration documentation and the storage-specific documentation for Local, S3, Google Cloud Storage, Azure, and HDFS to describe the new configuration property, supported formats, defaults, and uncompressed storage behavior. #### Release note Introduces the `druid.storage.compressionFormat` configuration for selecting the deep-storage segment compression format. Supports ZIP and uncompressed directory storage across Local, S3, Google Cloud Storage, and Azure, with additional LZ4 support for HDFS. The existing `druid.storage.zip` property remains available for backward compatibility but is deprecated. <hr> ##### Key changed/added classes in this PR - `DeepStorageSegmentConfig` - `CompressionUtils` - `LocalDataSegmentPusher` - `S3DataSegmentPusher` - `GoogleDataSegmentPusher` - `AzureDataSegmentPusher` - `HdfsDataSegmentPusherConfig` - `HdfsDataSegmentPusher` - `HdfsDataSegmentPuller` - `HdfsDataSegmentKiller` <hr> This PR has: - [ ] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [x] a release note entry in the PR description. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in `licenses.yaml`. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
