[
https://issues.apache.org/jira/browse/DRILL-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336950#comment-16336950
]
ASF GitHub Bot commented on DRILL-6080:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1090#discussion_r163456760
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/record/selection/SelectionVector4.java
---
@@ -31,8 +31,9 @@
private int length;
public SelectionVector4(ByteBuf vector, int recordCount, int
batchRecordCount) throws SchemaChangeException {
- if (recordCount > Integer.MAX_VALUE /4) {
- throw new SchemaChangeException(String.format("Currently, Drill can
only support allocations up to 2gb in size. You requested an allocation of %d
bytes.", recordCount * 4));
+ if (recordCount > Integer.MAX_VALUE / 4) {
+ throw new SchemaChangeException(String.format("Currently, Drill can
only support allocations up to 2gb in size. " +
+ "You requested an allocation of %d bytes.", recordCount * 4));
--- End diff --
This code was original, I only cleaned it up. It is true that the
`ValueVector` code will fail for an allocation above 2 GB. But, that code is
not used for the SV4. (Strangely, a selection *vector* is not a value
*vector*...) So, the SV4 must impose its own limit.
The limit is, essentially, 32K batches * 64K records per batch. (Given how
we do the math, it could be 64K batches since we mask off the high bits after
the shift in generated code.)
I'm guessing the original thinking was: we can allocate vectors up to 2 GB,
so we should only allow SV4s of the same size. With 4-bytes per SV4-entry, we
can allocate up to `(Integer.MAX_VALUE + 1) / 4` bytes.
Code like this is hard to reason about because 1) it is existing and 2) is
it broken. As we know, any allocation above 16 MB may fail due to memory
fragmentation, so allowing a 2 GB size is wildly optimistic on a busy system or
one that has been running long enough that direct memory all resides on the
Netty free list.
> Sort incorrectly limits batch size to 65535 records rather than 65536
> ---------------------------------------------------------------------
>
> Key: DRILL-6080
> URL: https://issues.apache.org/jira/browse/DRILL-6080
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.12.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Minor
> Fix For: 1.13.0
>
>
> Drill places an upper limit on the number of rows in a batch of 64K. That is
> 65,536 decimal. When we index records, the indexes run from 0 to 64K-1 or 0
> to 65,535.
> The sort code incorrectly uses {{Character.MAX_VALUE}} as the maximum row
> count. So, if an incoming batch uses the full 64K size, sort ends up
> splitting batches unnecessarily.
> The fix is to instead use the correct constant `ValueVector.MAX_ROW_COUNT`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
