andygrove opened a new issue, #386: URL: https://github.com/apache/datafusion-spark/issues/386
## Is your feature request related to a problem or challenge? Spark rewrites a `RuntimeReplaceable` expression into another expression before optimization. Functions here that are `RuntimeReplaceable` in Spark include `base64`, `encode`, `equal_null`, `ilike`, `luhn_check`, `make_time`, `parse_url`, `positive`, `time_trunc`, `to_binary`, `to_json`, and the `url_*` and `try_*` functions. Twelve functions here implement `ScalarUDFImpl::simplify` to rewrite themselves in the same way: `add_months`, `base64`, `date_diff`, `date_part`, `date_trunc`, `equal_null`, `if`, `nanvl`, `positive`, `time_trunc`, `trunc` and `unix_date`. Several of them have no working `invoke_with_args`, so they fail when they are evaluated without the simplifier, which happens in the sqllogictest evaluate mode or when a caller builds a physical plan directly (#305, #306, #307, #309, #311, #312, #313, #376). ## Describe the solution you'd like A shared helper for replaceable functions. A function would declare its replacement once, as a function from its argument expressions to an `Expr`, and the helper would provide: - `simplify`, which returns the replacement - `invoke_with_args`, which evaluates the same replacement over the arguments. For example, it could plan it as a physical expression over a batch of the argument values, and cache the plan per argument types. Each function would then behave the same whether or not the simplifier runs, and the `invoke_with_args` bugs above would be fixed together. ## Describe alternatives you've considered Write an `invoke_with_args` for each function, as the bug issues above suggest. That means implementing each function twice, and the two versions can drift apart. ## Additional context Spark 4.1: `RuntimeReplaceable` and `InheritAnalysisRules` in `Expression.scala`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
