[
https://issues.apache.org/jira/browse/ARROW-10165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Lamb updated ARROW-10165:
--------------------------------
Description:
When the DataFusion planner inserts casts, today it relies on special logic to
determine the valid coded casts.
The actual arrow cast kernels support a much wider range of data types, and
thus DataFusion is artificially limiting the casts it supports for no
particularly good reason I can see.
This ticket tracks the work to remove the extra casting checking in the
datafusion planner and instead simply rely on runtime check of arrow cast
compute kernel
The potential downside of this approach is that the error may be generated
later in the execution process (rather than the planner), and possibly have a
less specific error message, the upside is there is less code and we get
several conversions immediately (like timestamp predicate casting)
I also plan to add DictionaryArray support to the casting kernels and I would
like to avoid having to replicate some part of that logic in DataFusion
was:
There is code in the datafusion physical planner that coerces arguments to
compatible types for some expressions (e.g. for equals:
https://github.com/apache/arrow/blob/master/rust/datafusion/src/physical_plan/expressions.rs#L1153)
This code needs to be modified to understand dictionary types (so, for example
we can express a predicate like {{col1 = "foo"}}, where {{col1}} is a
DictionaryArray.
> [Rust] [DataFusion] Add DictionaryArray coercion support
> --------------------------------------------------------
>
> Key: ARROW-10165
> URL: https://issues.apache.org/jira/browse/ARROW-10165
> Project: Apache Arrow
> Issue Type: Sub-task
> Reporter: Andrew Lamb
> Assignee: Andrew Lamb
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When the DataFusion planner inserts casts, today it relies on special logic
> to determine the valid coded casts.
> The actual arrow cast kernels support a much wider range of data types, and
> thus DataFusion is artificially limiting the casts it supports for no
> particularly good reason I can see.
> This ticket tracks the work to remove the extra casting checking in the
> datafusion planner and instead simply rely on runtime check of arrow cast
> compute kernel
> The potential downside of this approach is that the error may be generated
> later in the execution process (rather than the planner), and possibly have a
> less specific error message, the upside is there is less code and we get
> several conversions immediately (like timestamp predicate casting)
> I also plan to add DictionaryArray support to the casting kernels and I would
> like to avoid having to replicate some part of that logic in DataFusion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
