helorymarceau opened a new pull request, #7694:
URL: https://github.com/apache/hop/pull/7694
In ParquetInput and ParquetInputMeta, the file size is retrieved as a long
but cast to int when passed to ByteArrayOutputStream:
long size = fileObject.getContent().getSize();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream((int) size);
When the file size exceeds Integer.MAX_VALUE (~2GB), the cast produces a
negative value, which is rejected by ByteArrayOutputStream.
Replaced the ByteArrayOutputStream based approach with a new ParquetStream
implementation that avoids loading the entire file into memory:
- Local files: uses Parquet's native LocalInputFile which reads directly
from disk via SeekableByteChannel with no size limit and minimal memory usage.
- Remote files (S3, MinIO...): implements a VfsSeekableInputStream that
reads via HopVFS and reopens the stream on each seek. This is functional for
all file sizes without memory constraints.
ParquetInput, ParquetInputData, ParquetInputMeta and ParquetTestUtil updated
to reflect the changes in ParquetStream.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [X] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [X] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [X] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [X] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [X] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
