dwsmith1983 commented on issue #2814: URL: https://github.com/apache/datafusion-comet/issues/2814#issuecomment-5637961390
The native Parquet writer now honors Spark's codec configuration. `CometDataWritingCommand` (spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala) resolves the codec with Spark's own precedence, the `compression` write option, then `parquet.compression`, then `spark.sql.parquet.compression.codec`, and accepts none/uncompressed, snappy, lz4, zstd and gzip (gzip landed last, in #4930, c65a5ee80); native/core/src/execution/operators/parquet_writer.rs maps each to the parquet-rs codec and carries the compression level for zstd. An lz4 level is not something parquet-rs exposes (`Compression::LZ4` takes no level), so that part of the ask has nothing to configure. Suggest closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
