jonasdedden commented on issue #10489: URL: https://github.com/apache/arrow-rs/issues/10489#issuecomment-5183349102
Hey @alamb, I think in general, I'd agree to your sentiment - the problem is just that this behavioral change is very silent and there are basically no guards from the `parquet` crate for people who don't know that they specifically have to tune the data page size limit. That potentially also could be a pretty niche config setting for most people, I at least never had to tune it before. For context, I stumbled over this issue when I was running a test system for a large internal library and just wanted to update some dependencies. That suddenly OOMed my machine (because test data was written to `/tmp` which happened to be a tmpfs volume). Note that the degenerated parquet file basically has size of `num_rows` * 8MiB, which can of course be troublesome for very high `num_rows`. The change proposed in https://github.com/apache/arrow-rs/pull/10505 looks interesting, but I'm not really an expert on this topic to accurately judge what to do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
