jorisvandenbossche commented on issue #40658: URL: https://github.com/apache/arrow/issues/40658#issuecomment-2009120646
It's not built in into the csv writer itself (for the reading side it is), but you can let it compress on the fly using `CompressedOutputStream`. See the usage example at https://arrow.apache.org/docs/python/csv.html#usage: ```python import pyarrow as pa import pyarrow.csv as csv with pa.CompressedOutputStream("tips.csv.gz", "gzip") as out: csv.write_csv(table, out) ``` That's only the direct CSV writer, and doesn't work through `pyarrow.dataset`. For compressed CSV in writing datasets we have an open feature request: https://github.com/apache/arrow/issues/34085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
