Smotrov commented on code in PR #18954:
URL: https://github.com/apache/datafusion/pull/18954#discussion_r2567581119
##########
datafusion/common/src/config.rs:
##########
@@ -2786,6 +2787,14 @@ config_namespace! {
/// The default behaviour depends on the
`datafusion.catalog.newlines_in_values` setting.
pub newlines_in_values: Option<bool>, default = None
pub compression: CompressionTypeVariant, default =
CompressionTypeVariant::UNCOMPRESSED
+ /// Compression level for the output file. The valid range depends on
the
+ /// compression algorithm:
+ /// - ZSTD: 1 to 22 (default: 3)
+ /// - GZIP: 0 to 10 (default: varies by implementation)
Review Comment:
Good catch @Jefffrey ! You're right that "varies by implementation" is
vague. Let me clarify:
The GZIP compression in async-compression uses flate2 under the hood.
Looking at the flate2 source code, the default is level 6:
```Rust
// From https://github.com/rust-lang/flate2-rs/blob/main/src/lib.rs#L220-L224
impl Default for Compression {
fn default() -> Compression {
Compression(6)
}
}
```
This is the standard zlib/gzip default (going back to the original zlib
implementation). The valid range is 0-9 (not 0-10 as I incorrectly wrote).
I'll update the comment to be more precise:
```Rust
/// - GZIP: 0 to 9 (default: 6)
```
The reason I was initially cautious is that some compression **libraries
allow you to swap backends** (e.g., flate2 can use miniz_oxide, zlib-rs, or
native zlib), but they all follow the same 0-9 range and default to 6 for
compatibility.
Would you like me to also update the code to fix the comment and rerun CI
workflows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
