pitrou commented on PR #36286: URL: https://github.com/apache/arrow/pull/36286#issuecomment-1609936769
> So our files end with hundred or even thousands of RowGroups, which causing Metadata grows huge. Hundres of row groups for which file size? > This patch exports `parquet::arrow::FileWriter::WriteBatch`, which will buffer the input in one RowGroup. Comparing to direct buffer the records in memory or in arrow file, buffering to parquet might be more straightforward @pitrou But does it increase performance? Decrease file size? By how much? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
