REASY commented on issue #4973: URL: https://github.com/apache/arrow-rs/issues/4973#issuecomment-1774068411
> What happens if you instead lower the write batch size to produce smaller pages, that way you may be able to raise the row group limit? > > As an aside 5MB rows is likely to stress most analytics systems, you may be better off storing point clouds separately... The typical expectation is for pages to be ~1MB let alone rows... - With `set_max_row_group_size(100)` and `.set_write_batch_size(10)` getting maximum resident set size = 5.31 Gbytes - With `set_max_row_group_size(1000)` and `.set_write_batch_size(10)` getting maximum resident set size = 5.65 Gbytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
