Liya Fan created ARROW-7746:
-------------------------------
Summary: [Java] Support large buffer for IPC
Key: ARROW-7746
URL: https://issues.apache.org/jira/browse/ARROW-7746
Project: Apache Arrow
Issue Type: Task
Components: Java
Reporter: Liya Fan
The motivation is described in
https://github.com/apache/arrow/pull/6323#issuecomment-580137629.
When the size of the ArrowBuf exceeds 2GB, our flighing library does not work
due to integer overflow.
This is because internally, we have used some data structures which are based
on 32-bit integers. To resolve the problem, we must revise/replace the data
structures to make them support 64-bit integers.
As a concrete example, we can see that when the server sends data through IPC,
an org.apache.arrow.flight.ArrowMessage object is created, and is wrapped as an
InputStream through the `asInputStream` method. In this method, we use data
stuctures like java.io.ByteArrayOutputStream and io.netty.buffer.ByteBuf, which
are based on 32-bit integers (we can observe that NettyArrowBuf#length and
ByteArrayOutputStream#count are both 32-bit integers).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
