Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22705#discussion_r224977082
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
@@ -195,7 +196,11 @@ object ChunkedByteBuffer {
val is = new FileInputStream(file)
ByteStreams.skipFully(is, offset)
val in = new LimitedInputStream(is, length)
- val chunkSize = math.min(maxChunkSize, length).toInt
+ // Though in theory you should be able to index into an array of size
Int.MaxValue, in practice
+ // jvms don't let you go up to limit. It seems you may only need - 2,
but we leave a little
+ // extra room.
+ val maxArraySize = Int.MaxValue - 512
--- End diff --
We already had seen the similar problem in other places. Can we use [this
value](https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java#L52)
here, too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
