roshiiiz commented on PR #6811: URL: https://github.com/apache/texera/pull/6811#issuecomment-5073605400
> If you want a faster fail path, then I wonder why we have to run the operator to check the file size. Did you consider a way to check the file size before running the operator? Also, exceeding 100MB doesn't necessarily mean the operator will cause an OOM. Can you reconsider the idea? That is a great suggestion, but checking the file size upfront introduces some edge cases. For instance, if a user uploads a .zip file, we cannot determine the uncompressed size of the extracted entries until the operator is actively streaming and unzipping them on the fly. (I might be overthinking this as well) However, I completely agree that your original suggestion is 100% the cleanest and best approach. I will go ahead and remove the threshold entirely. I'll update the PR to let it hit the Java array limit, and we can simply catch the resulting OutOfMemoryError or IllegalArgumentException and wrap it in user-friendly large-binary hint. Thanks for the feedback I will update the PR with this approach shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
