Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22705#discussion_r224965160
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
@@ -195,7 +196,11 @@ object ChunkedByteBuffer {
val is = new FileInputStream(file)
ByteStreams.skipFully(is, offset)
val in = new LimitedInputStream(is, length)
- val chunkSize = math.min(maxChunkSize, length).toInt
+ // Though in theory you should be able to index into an array of size
Int.MaxValue, in practice
+ // jvms don't let you go up to limit. It seems you may only need - 2,
but we leave a little
+ // extra room.
+ val maxArraySize = Int.MaxValue - 512
--- End diff --
Looks like the max is safely more like `Int.MaxValue - 5` from things like
https://www.quora.com/What-is-the-maximum-size-of-the-array-in-Java You could
probably push it further, but whatever.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
