alamb commented on issue #15028: URL: https://github.com/apache/datafusion/issues/15028#issuecomment-2704750842
> With the default settings, the parallel parquet encoding will use 1 rowgroup writer and you will be buffering uncompressed data pages for that rowgroup. Have you tried to decrease the [page size](https://github.com/apache/datafusion/blob/34efd1fbae39eb0441a43ab976fc23001d1f674a/datafusion/common/src/config.rs#L514)? I am not sure if just the page size is enough to reduce the memory needed -- you might also have to reduce the max row group size: https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_max_row_group_size -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
