kosiew commented on code in PR #981:
URL: https://github.com/apache/datafusion-python/pull/981#discussion_r1898185864
##########
python/datafusion/dataframe.py:
##########
@@ -620,16 +620,24 @@ def write_csv(self, path: str | pathlib.Path,
with_header: bool = False) -> None
def write_parquet(
self,
path: str | pathlib.Path,
- compression: str = "uncompressed",
+ compression: str = "ZSTD",
compression_level: int | None = None,
) -> None:
"""Execute the :py:class:`DataFrame` and write the results to a
Parquet file.
Args:
- path: Path of the Parquet file to write.
- compression: Compression type to use.
- compression_level: Compression level to use.
- """
+ path (str | pathlib.Path): The file path to write the Parquet file.
+ compression (str): The compression algorithm to use. Default is "ZSTD".
+ compression_level (int | None): The compression level to use. For
ZSTD, the
+ recommended range is 1 to 22, with the default being 3. Higher
levels
+ provide better compression but slower speed.
+ """
+ # default compression level to 3 for ZSTD
+ if compression == "ZSTD":
+ if compression_level is None:
+ compression_level = 3
Review Comment:
> 3 seems like an awfully low compression default. We should evaluate what
other libraries use as the default compression setting.
I used the default compression level in manual from Facebook (author of
zstd) - https://facebook.github.io/zstd/zstd_manual.html
I could not find a default in DuckDB's documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
