zabetak commented on issue #1404: URL: https://github.com/apache/orc/issues/1404#issuecomment-1439889316
At first glance 2GB of metadata seems big especially when considering the toy example that I made in ORC-1361. However, if you have a 500GB ORC file then 2GB of metadata does not appear too big anymore so things are relative. Are there limitations on the maximum size of an ORC file? Do we support such kind of use-cases? If we add a limit while writing (which by the way I also suggested in https://github.com/protocolbuffers/protobuf/issues/11729) then we should define what happens when the limit is exceeded: * drop all metadata * fail the entire write * ignore metadata over the limit (keep partial metadata) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
