eadwright edited a comment on pull request #902: URL: https://github.com/apache/parquet-mr/pull/902#issuecomment-832990874
> Can you come up with a unit test? I want to, and I'm open to ideas. To re-create the problem you need a parquet file which has a row group over 2GB in size, which translates to way over 2GB of memory used. Perhaps 10x more - I've not investigated which field types may work the best. My concern is whether such a test would blow the constraints during CI or put an unreasonable memory burden on anyone wanting to build/test the project. What are reasonable limits there? I will add no tests have found a regression with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
