[ 
https://issues.apache.org/jira/browse/ARROW-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156928#comment-16156928
 ] 

Wes McKinney commented on ARROW-1463:
-------------------------------------

[~gortizja] this is probably a discussion for a whole separate JIRA. A couple 
points:

> Non nullable structures doesn't return nulls, which can be used by some 
> static checkers or languages like Kotlin.

This is true, hypothetically, but I question whether this benefit can only be 
reaped by doubling the size of the type hierarchy. In the instances where you 
can choose an optimized code path for non-null data, you could elect to invoke 
that same code path when the null count is zero (which you will want to do 
regardless of the nullability). This is sort of a "composition over 
inheritance" argument -- it's not to say that we can have some helper 
interfaces that specialize based on metadata, but as a property of the type 
hierarchy I am not sure if it's worth the increased code size. 

> So this is a point on having two implementations: one with nulls and one 
> without them.

Since you can branch at the vector level, the performance implications are less 
obvious (benchmarks could help settle it). Somewhere you must dynamic dispatch 
to the correct kernel function for some analytical operation, and the dispatch 
could be based on null count. 

> [if] the nullable is just a decoration/delegation on the non-nullable, 
> nullable structures may behave quite worst than we expect for the same reason

The performance issues you cite would appear mostly in non-vectorized code. If 
you're doing something performance sensitive, you're more likely to be working 
directly with the buffers, and selecting kernel functions based on the dynamic 
metadata (which can include nullability / null count). So if you implement Sum, 
you will want to have a sum with nulls and a sum without. Somewhere you have to 
decide which one to call. You will want to elect to call the SumWithoutNulls 
when you have a nullable vector but with zero null count anyway, so this 
actually makes your kernel dispatch more complex. 

> [JAVA] Restructure ValueVector hierarchy to minimize compile-time generated 
> code
> --------------------------------------------------------------------------------
>
>                 Key: ARROW-1463
>                 URL: https://issues.apache.org/jira/browse/ARROW-1463
>             Project: Apache Arrow
>          Issue Type: Improvement
>            Reporter: Jacques Nadeau
>            Assignee: SIDDHARTH TEOTIA
>
> The templates used in the java package are very high mainteance and the if 
> conditions are hard to track. As started in the discussion here: 
> https://github.com/apache/arrow/pull/1012, I'd like to propose that we modify 
> the structure of the internal value vectors and code generation dynamics.
> Create new abstract base vectors:
> BaseFixedVector
> BaseVariableVector
> BaseNullableVector
> For each of these, implement all the basic functionality of a vector without 
> using templating.
> Evaluate whether to use code generation to generate specific specializations 
> of this functionality for each type where needed for performance purposes 
> (probably constrained to mutator and accessor set/get methods). Giant and 
> complex if conditions in the templates are actually worse from my perspective 
> than a small amount of hand written duplicated code since templates are much 
> harder to work with. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to