[
https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069341#comment-16069341
]
ASF GitHub Bot commented on DRILL-5517:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/840#discussion_r124938564
--- Diff: exec/vector/src/main/codegen/templates/NullableValueVectors.java
---
@@ -674,6 +764,14 @@ public void reset(){
setCount = 0;
<#if type.major = "VarLen">lastSet = -1;</#if>
}
+
+ @Override
+ public void exchange(ValueVector.Mutator other) {
--- End diff --
Yes, indeed I am trusting the caller, in the same way we trust the caller
to free a vector's buffer at the right time, to call `setSafe()` instead of
`set()`, to write to the vector only once, and so on. In short, if someone does
try to use `exchange()` with a vector owned by another allocator, the allocator
is likely to complain about memory leaks.
An alternative is to always use transfer pairs, though that adds quite a
bit of complexity for this one use case.
This method has a very specific usage: an operator writes to a vector,
discovers that the vector overflows, and must swap buffers between two sets of
vectors. This swapping is necessary because downstream operators depend on a
vector instance, as does the writer, so it is necessary to swap buffers into
and out of these fixed set of value vectors. Rather confusing, but the most
performant solution given how vectors work today.
For this particular method, confusion is understandable. This is an
`exchange()` on the `Mutator` class, swapping the state that nullable vector
mutators maintain. It is called from a vector `exchange()` method earlier in
this file. Added a comment to help future readers.
> Provide size-aware set operations in value vectors
> --------------------------------------------------
>
> Key: DRILL-5517
> URL: https://issues.apache.org/jira/browse/DRILL-5517
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> DRILL-5211 describes a memory fragmentation issue in Drill. The resolution is
> to limit vector sizes to 16 MB (the size of Netty memory allocation "slabs.")
> Effort starts by providing "size-aware" set operations in value vectors which:
> * Operate as {{setSafe()}} while vectors are below 16 MB.
> * Throw a new, specific exception ({{VectorOverflowException}}) if setting
> the value (and growing the vector) would exceed the vector limit.
> The methods in value vectors then become the foundation on which we can
> construct size-aware record batch "writers."
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
