andygrove opened a new issue, #386:
URL: https://github.com/apache/datafusion-spark/issues/386

   ## Is your feature request related to a problem or challenge?
   
   Spark rewrites a `RuntimeReplaceable` expression into another expression 
before optimization. Functions here that are `RuntimeReplaceable` in Spark 
include `base64`, `encode`, `equal_null`, `ilike`, `luhn_check`, `make_time`, 
`parse_url`, `positive`, `time_trunc`, `to_binary`, `to_json`, and the `url_*` 
and `try_*` functions.
   
   Twelve functions here implement `ScalarUDFImpl::simplify` to rewrite 
themselves in the same way: `add_months`, `base64`, `date_diff`, `date_part`, 
`date_trunc`, `equal_null`, `if`, `nanvl`, `positive`, `time_trunc`, `trunc` 
and `unix_date`. Several of them have no working `invoke_with_args`, so they 
fail when they are evaluated without the simplifier, which happens in the 
sqllogictest evaluate mode or when a caller builds a physical plan directly 
(#305, #306, #307, #309, #311, #312, #313, #376).
   
   ## Describe the solution you'd like
   
   A shared helper for replaceable functions. A function would declare its 
replacement once, as a function from its argument expressions to an `Expr`, and 
the helper would provide:
   
   - `simplify`, which returns the replacement
   - `invoke_with_args`, which evaluates the same replacement over the 
arguments. For example, it could plan it as a physical expression over a batch 
of the argument values, and cache the plan per argument types.
   
   Each function would then behave the same whether or not the simplifier runs, 
and the `invoke_with_args` bugs above would be fixed together.
   
   ## Describe alternatives you've considered
   
   Write an `invoke_with_args` for each function, as the bug issues above 
suggest. That means implementing each function twice, and the two versions can 
drift apart.
   
   ## Additional context
   
   Spark 4.1: `RuntimeReplaceable` and `InheritAnalysisRules` in 
`Expression.scala`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to