andygrove opened a new issue, #349:
URL: https://github.com/apache/datafusion-spark/issues/349

   ### Is your feature request related to a problem or challenge?
   
   Spark has no dictionary or run-end-encoded types, but DataFusion and Comet 
pass them to functions: a Parquet string column is often read as a dictionary. 
Functions here handle them unevenly. The shared common-type rules in 
`function/type_coercion.rs` (#345) now give a dictionary argument the type of 
its values, so `array_position`, `array_remove`, `array_insert`, `greatest` and 
`least` accept one and cast it to the value type. Other functions handle them 
on their own: `dayofweek` maps the dictionary values and `hash` has a 
dictionary path, while others reject them. Run-end-encoded arguments are not 
handled in the shared rules at all.
   
   ### Describe the solution you'd like
   
   Decide how the crate should treat these encodings, and apply it 
consistently. Questions to settle:
   
   - Should coercion keep the encoding where a function can work on it directly 
(for example, compare or hash the dictionary values once and map the keys), 
instead of casting to the value type?
   - Should run-end-encoded arguments be accepted the same way as dictionaries?
   - Should this be part of the shared input-type coercion proposed in #346, so 
each function does not handle encodings itself?
   - What do Comet's call paths need, since Comet calls `invoke_with_args` 
directly without coercion?
   
   ### Describe alternatives you've considered
   
   Keep handling encodings per function, as now.
   
   ### Additional context
   
   Raised while moving `array_insert` onto the shared rules in #70, which had 
unwrapped dictionaries itself.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to