Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1090#discussion_r163456760
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java
---
@@ -31,8 +31,9 @@
private int length;
public SelectionVector4(ByteBuf vector, int recordCount, int
batchRecordCount) throws SchemaChangeException {
- if (recordCount > Integer.MAX_VALUE /4) {
- throw new SchemaChangeException(String.format("Currently, Drill can
only support allocations up to 2gb in size. You requested an allocation of %d
bytes.", recordCount * 4));
+ if (recordCount > Integer.MAX_VALUE / 4) {
+ throw new SchemaChangeException(String.format("Currently, Drill can
only support allocations up to 2gb in size. " +
+ "You requested an allocation of %d bytes.", recordCount * 4));
--- End diff --
This code was original, I only cleaned it up. It is true that the
`ValueVector` code will fail for an allocation above 2 GB. But, that code is
not used for the SV4. (Strangely, a selection *vector* is not a value
*vector*...) So, the SV4 must impose its own limit.
The limit is, essentially, 32K batches * 64K records per batch. (Given how
we do the math, it could be 64K batches since we mask off the high bits after
the shift in generated code.)
I'm guessing the original thinking was: we can allocate vectors up to 2 GB,
so we should only allow SV4s of the same size. With 4-bytes per SV4-entry, we
can allocate up to `(Integer.MAX_VALUE + 1) / 4` bytes.
Code like this is hard to reason about because 1) it is existing and 2) is
it broken. As we know, any allocation above 16 MB may fail due to memory
fragmentation, so allowing a 2 GB size is wildly optimistic on a busy system or
one that has been running long enough that direct memory all resides on the
Netty free list.
---
