bersprockets commented on code in PR #37852:
URL: https://github.com/apache/spark/pull/37852#discussion_r970067289
##########
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeArrayWriter.java:
##########
@@ -55,10 +55,19 @@ public void initialize(int numElements) {
this.startingOffset = cursor();
+ long fixedPartInBytesLong =
+ ByteArrayMethods.roundNumberOfBytesToNearestWord((long) elementSize *
numElements);
+ long totalInitialSize = headerInBytes + fixedPartInBytesLong;
+
+ if (totalInitialSize > Integer.MAX_VALUE) {
+ throw new IllegalArgumentException(
Review Comment:
Just a point of clarification. The issue isn't too many elements per se.
It's too many elements for the given element size.
If, in the above example, I had cast `val` as int, the `collect_list` would
have succeeded (i.e., 268271216 int elements is just fine, but the same number
of bigint elements is not).
How about this for a message?:
"Cannot initialize array with %numElements elements of size %size"
Caveat: my suggested message will be a little confusing for arrays of
non-primitives, where the element size is always 8 (for the size/offset).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
