aglinxinyuan commented on PR #6811: URL: https://github.com/apache/texera/pull/6811#issuecomment-5073863181
> > If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary? > > > > Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit. > > > > The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure). > > > > However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR. Do you have some data to support this? For example, can you test the maximum possible file size for this operator? We want to set this limit to upper bound instead of lower bound. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
