gszadovszky commented on pull request #902: URL: https://github.com/apache/parquet-mr/pull/902#issuecomment-841139504
@advancedxy, thanks for explaining. I think, the best option is 2. It is up to the user to provide enough resources for handling the large row groups or not writing them. Meanwhile, even though I've written I can accept lacking of unit tests in some situations my concern in this case is I am not sure that every aspect of a large row group is handled properly. So, we clearly need to validate this fix with such large row groups. This test can be even implemented in this source code but we must not include it in the unit tests or integration tests we run regularly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
