[
https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013037#comment-16013037
]
Paul Rogers commented on DRILL-5517:
------------------------------------
Code reviewers can't help but notice the level of redundant code that has crept
into value vectors. We now have three distinct ways to set values:
* {{set()}} -- set values, throws an exception if vector is full, does not
expand vector
* {{setSafe()}} -- always expands vectors to fit values, with no size limits
* {{setScalar()}}, {{setArrayItem()}} -- sets values, expands vectors up to a
limit, throws an exception after that
Then, many vectors provide multiple ways to set the data: from a {{DrillBuf}},
from a byte array, from a scalar, from an Object.
The result is the proliferation of very similar code. Over the long term, we
may want to control the redundancy.
For now, existing code uses all the various methods. To avoid breaking existing
code, the new methods are added in parallel. As code is converted to be
size-aware, we have an opportunity to retire the other two variations.
> Provide size-aware set operations in value vectors
> --------------------------------------------------
>
> Key: DRILL-5517
> URL: https://issues.apache.org/jira/browse/DRILL-5517
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is
> to limit vector sizes to 16 MB (the size of Netty memory allocation "slabs.")
> Effort starts by providing "size-aware" set operations in value vectors which:
> * Operate as {{setSafe()}} while vectors are below 16 MB.
> * Throw a new, specific exception ({{VectorOverflowException}}) if setting
> the value (and growing the vector) would exceed the vector limit.
> The methods in value vectors then become the foundation on which we can
> construct size-aware record batch "writers."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
