Github user liufengdb commented on a diff in the pull request:
https://github.com/apache/spark/pull/19266#discussion_r143346155
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java
---
@@ -35,6 +35,11 @@
* if the fields of row are all fixed-length, as the size of result row is
also fixed.
*/
public class BufferHolder {
+
+ // Some JVMs can't allocate arrays of length Integer.MAX_VALUE; actual
max is somewhat
+ // smaller. Be conservative and lower the cap a little.
+ private static final int ARRAY_MAX = Integer.MAX_VALUE - 8;
--- End diff --
@srowen I think we can use `Integer.MAX_VALUE - 7` instead of
`Integer.MAX_VALUE - 8` to make the size align with words, otherwise, this
check will fail:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java#L170.
This is the reason why all the size inputs to the methods are rounded, for
example,
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java#L216.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
