[
https://issues.apache.org/jira/browse/ARROW-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208691#comment-17208691
]
Andrew Lamb commented on ARROW-8464:
------------------------------------
FYI [~andygrove] -- I am doing some part of this in ARROW-10159 -- however, the
initial implementation effectively converts DictionaryArray --> PrimitiveArray
/ StringArray and then uses the existing processing.
To support the actual efficient processing usecase, I think significant work
will be needed to add appropriate dictionary support to the arrow compute
kernels
> [Rust] [DataFusion] Add support for dictionary types
> ----------------------------------------------------
>
> Key: ARROW-8464
> URL: https://issues.apache.org/jira/browse/ARROW-8464
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust - DataFusion
> Reporter: Andy Grove
> Priority: Major
>
> Usecases: Efficiently process large columns of low cardinality Strings
>
> * BatchIterator should accept both DictionaryBatch and RecordBatch
> * Type Coercion optimizer rule should inject expression for converting
> dictionary value types to index types (for equality expressions, and
> IN(values, ...)
> * Physical expression would lookup index for dictionary values referenced in
> the query so that at runtime, only indices are being compared per batch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
