Seminko commented on issue #47133: URL: https://github.com/apache/arrow/issues/47133#issuecomment-5078319270
Hey @GujaLomsadze thanks for the investigation and getting back to me! As you laid things out it makes senso to me and IMO things are working as intended. This bothered me because I did not know what and where the issue was. The exception `malloc of size n failed` was not helpful either. It's been a long time so I no longer have it in memory. My instinct tells me is if we are able to tell early that the memory is going to balloon up "uncontrollably" before it actually does, we should let the user know to use a schema maybe as a warning? If we are not able to tell early, the info should be somewhat prominent in the documentation. Either way, maybe even catch `malloc of size n failed`, check for the number of cells, and if it's above a threshold throw a more useful exception instead. Anything that would make the user understand where the problem is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
