Github user buryat commented on a diff in the pull request:
https://github.com/apache/spark/pull/19266#discussion_r139616165
--- Diff:
core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala ---
@@ -126,22 +126,20 @@ private[spark] class CompactBuffer[T: ClassTag]
extends Seq[T] with Serializable
/** Increase our size to newSize and grow the backing array if needed. */
private def growToSize(newSize: Int): Unit = {
- if (newSize < 0) {
- throw new UnsupportedOperationException("Can't grow buffer past
Int.MaxValue elements")
+ val arrayMax = Int.MaxValue - 8
+ if (newSize < 0 || newSize - 2 > arrayMax) {
+ throw new UnsupportedOperationException(s"Can't grow buffer past
$arrayMax elements")
}
val capacity = if (otherElements != null) otherElements.length + 2
else 2
if (newSize > capacity) {
- var newArrayLen = 8
+ var newArrayLen = 8L
while (newSize - 2 > newArrayLen) {
--- End diff --
ah, I see that it's reserved, wasn't clear to me why `- 2`, seems like a
magic number to me, I see it other places too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
