twest820 commented on issue #37069: URL: https://github.com/apache/arrow/issues/37069#issuecomment-1794979738
Agree with @pitrou on testing and migrating to 64 bit sizes and offsets. From a look at the PR diffs I'd anticipate trouble at `offset` and `paddedLength` since those remain 32 bit—a signed 32 bit byte count on a >2 GB file is most likely a defect (barring relative offsets used in special cases). Since C#'s maximum object length is `Int32.MaxValue`, code paths using Flatbuf.`RecordBatch` can't exceed 2 GB [as it's 1:1](https://github.com/apache/arrow/blob/main/csharp/src/Apache.Arrow/Flatbuf/RecordBatch.cs) with `ByteBuffer`. If Flatbuf use isn't congruent with `IEnumerable<ArrowArray>` on the public `RecordBatch` that might another area to look for trouble. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
