Jefffrey commented on code in PR #18954:
URL: https://github.com/apache/datafusion/pull/18954#discussion_r2567110618
##########
datafusion/common/src/file_options/csv_writer.rs:
##########
@@ -41,6 +43,20 @@ impl CsvWriterOptions {
Self {
writer_options,
compression,
+ compression_level: None,
+ }
+ }
+
+ /// Create a new `CsvWriterOptions` with the specified compression level.
+ pub fn new_with_level(
+ writer_options: WriterBuilder,
+ compression: CompressionTypeVariant,
+ compression_level: Option<u32>,
Review Comment:
Is it better to just have `compression_level: u32` and direct users to use
`new` if they want default (`None`) compression level? Thoughts? 🤔
##########
datafusion/common/src/config.rs:
##########
@@ -2786,6 +2787,14 @@ config_namespace! {
/// The default behaviour depends on the
`datafusion.catalog.newlines_in_values` setting.
pub newlines_in_values: Option<bool>, default = None
pub compression: CompressionTypeVariant, default =
CompressionTypeVariant::UNCOMPRESSED
+ /// Compression level for the output file. The valid range depends on
the
+ /// compression algorithm:
+ /// - ZSTD: 1 to 22 (default: 3)
+ /// - GZIP: 0 to 10 (default: varies by implementation)
Review Comment:
What does `varies by implementation` mean here? Depends on system library,
depends on rust crate dependency (in which case ideally we'd know which it is)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
