FrankChen021 opened a new issue, #20767: URL: https://github.com/apache/druid/issues/20767
### Motivation Deep storage segment packaging is controlled by two different properties today: - `druid.storage.zip` (`true`/`false`): local, S3, Azure, Google. - `druid.storage.compressionFormat` (`zip`/`lz4`): HDFS only. They express the same idea: how a segment directory is packaged when pushed to deep storage. The two properties have drifted apart, so operators need to know which property each deep storage reads. ### History | PR | Release | Change | |---|---|---| | #13394 | 26.0.0 | `druid.storage.zip` added for local deep storage (default `false`). HDFS not covered. | | #18544 | 35.0.0 | S3 honors `druid.storage.zip` (default `true`). | | #18982 | 37.0.0 | `druid.storage.compressionFormat` added for HDFS (`zip`, `lz4`; default `zip`). HDFS still cannot store segments unarchived. | | #20327 | unreleased | `DeepStorageSegmentConfig` introduced; Azure honors `druid.storage.zip`. | | #20382 | unreleased | Google honors `druid.storage.zip`. | This was discussed in https://github.com/apache/druid/pull/20382#issuecomment-5765658570, where adding `druid.storage.compressionFormat` to `DeepStorageSegmentConfig` was considered reasonable. ### Proposal 1. Move `compressionFormat` into `DeepStorageSegmentConfig` so all deep storages read the same property. 2. Add a `none` value: segment files are pushed as-is, without an archive. 3. Deprecate `druid.storage.zip` as an alias: `true` → `zip`, `false` → `none`. Fail at startup if both are set and conflict. 4. Keep current defaults: `none` for local, `zip` for S3, Azure, Google and HDFS. 5. Each deep storage validates the values it supports at startup. For example, `lz4` remains HDFS-only, and `gz`/`xz`/etc. are rejected rather than failing at push time. 6. HDFS gains `none`: push unarchived files, and let the killer delete them (the puller already handles directories). ### Compatibility - No behavior change unless the configuration is changed. - Existing segments keep loading: the puller infers the format from the load spec path (`.zip`, `.lz4`, or a directory), so mixed packaging in one datasource works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
