[
https://issues.apache.org/jira/browse/DRILL-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009342#comment-17009342
]
Weijie Tong commented on DRILL-7087:
------------------------------------
Hi [~IhorHuzenko] , I have done this work for a long time. But since my job
role has turned from Drill, I have no time to continue this work at the
community. So hope someone could take this work, and sorry for that.
Your work is great. I think Drill should turn to Arrow to depend on a continual
tuned memory format project. My solution is not a perfect final solution. We
need to pay for the cost to count for the null tag bit to a nullable vector.
Completely turning Drill VV to Arrow is a large job to finish and review. Maybe
an adapter to turn Drill's VV to Arrow's is needed. So we could finish this job
one by one. But that's still a tough job, as you need to rewrite all the
operators, storage plugins, and udfs.
> Integrate Arrow's Gandiva into Drill
> ------------------------------------
>
> Key: DRILL-7087
> URL: https://issues.apache.org/jira/browse/DRILL-7087
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Codegen, Execution - Relational Operators
> Reporter: Weijie Tong
> Assignee: Weijie Tong
> Priority: Major
>
> It's a prior work to integrate arrow into drill by invoking the its gandiva
> feature. Comparing arrow and drill 's in memory column representation ,
> there's different null representation internal now. Drill use 1 byte while
> arrow using 1 bit to indicate one null row. Also all columns of arrow is
> nullable now. Apart from those basic differences , they have same memory
> representation to the different data types.
> The integrating strategy is to invoke arrow's JniWrapper's native method
> directly by passing the ValueVector's memory address.
> I have done a implementation at our own Drill version by integrating gandiva
> into Drill's project operator. The performance shows that there's nearly 1
> times performance gain at expression computation.
> So if there's no objection , I will submit a related PR to contribute this
> feature. Also this issue waits for arrow's related issue[ARROW-4819].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
