[
https://issues.apache.org/jira/browse/ARROW-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156928#comment-16156928
]
Wes McKinney commented on ARROW-1463:
-------------------------------------
[~gortizja] this is probably a discussion for a whole separate JIRA. A couple
points:
> Non nullable structures doesn't return nulls, which can be used by some
> static checkers or languages like Kotlin.
This is true, hypothetically, but I question whether this benefit can only be
reaped by doubling the size of the type hierarchy. In the instances where you
can choose an optimized code path for non-null data, you could elect to invoke
that same code path when the null count is zero (which you will want to do
regardless of the nullability). This is sort of a "composition over
inheritance" argument -- it's not to say that we can have some helper
interfaces that specialize based on metadata, but as a property of the type
hierarchy I am not sure if it's worth the increased code size.
> So this is a point on having two implementations: one with nulls and one
> without them.
Since you can branch at the vector level, the performance implications are less
obvious (benchmarks could help settle it). Somewhere you must dynamic dispatch
to the correct kernel function for some analytical operation, and the dispatch
could be based on null count.
> [if] the nullable is just a decoration/delegation on the non-nullable,
> nullable structures may behave quite worst than we expect for the same reason
The performance issues you cite would appear mostly in non-vectorized code. If
you're doing something performance sensitive, you're more likely to be working
directly with the buffers, and selecting kernel functions based on the dynamic
metadata (which can include nullability / null count). So if you implement Sum,
you will want to have a sum with nulls and a sum without. Somewhere you have to
decide which one to call. You will want to elect to call the SumWithoutNulls
when you have a nullable vector but with zero null count anyway, so this
actually makes your kernel dispatch more complex.
> [JAVA] Restructure ValueVector hierarchy to minimize compile-time generated
> code
> --------------------------------------------------------------------------------
>
> Key: ARROW-1463
> URL: https://issues.apache.org/jira/browse/ARROW-1463
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Jacques Nadeau
> Assignee: SIDDHARTH TEOTIA
>
> The templates used in the java package are very high mainteance and the if
> conditions are hard to track. As started in the discussion here:
> https://github.com/apache/arrow/pull/1012, I'd like to propose that we modify
> the structure of the internal value vectors and code generation dynamics.
> Create new abstract base vectors:
> BaseFixedVector
> BaseVariableVector
> BaseNullableVector
> For each of these, implement all the basic functionality of a vector without
> using templating.
> Evaluate whether to use code generation to generate specific specializations
> of this functionality for each type where needed for performance purposes
> (probably constrained to mutator and accessor set/get methods). Giant and
> complex if conditions in the templates are actually worse from my perspective
> than a small amount of hand written duplicated code since templates are much
> harder to work with.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)