roshiiiz commented on PR #6811: URL: https://github.com/apache/texera/pull/6811#issuecomment-5072105008
> Do you think producing OOM is a bug? I think it's natural to produce OOM error if the operator doesn't support reading more than 2GB file. I am okay if the direction is to make the error message user-friendly and guide to using `large-binary` type but I don't think introducing a threshold is a good solution. I completely understand your point, and I agree that an OOM is natural when hitting that 2GB limit. The main reason I initially leaned toward a threshold was actually to solve a user experience issue. The original use case that sparked this involved someone uploading a 9GB file. They ended up waiting a really long time for it to process, only for the whole thing to crash at the very end with a generic error. My goal was to introduce a way to 'fail fast' so a new user unaware of the limit doesn't waste 30 minutes waiting on a 10GB file just to get an OOM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
