voonhous commented on code in PR #19853:
URL: https://github.com/apache/hudi/pull/19853#discussion_r3968381022
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedureFilterUtils.scala:
##########
@@ -389,6 +416,73 @@ object HoodieProcedureFilterUtils {
}
}
+ // Resolves a function not covered by the hardcoded table above via Spark's
own FunctionRegistry.
+ // A resolved result is only usable if it can actually be eval()'d one row
at a time, which
+ // several categories of otherwise-valid expressions cannot:
RuntimeReplaceable placeholders
+ // (nvl, ifnull, left, right) need substitution the analyzer normally
performs but skips here,
+ // and can themselves unwrap to another RuntimeReplaceable (regexp_substr ->
NullIf) so the
+ // unwrap has to run to a fixed point; aggregates (percentile, collect_list)
only make sense
+ // across real aggregation; generators (explode, inline) only work inside a
projection;
+ // non-deterministic functions (rand, uuid, spark_partition_id) expect
per-partition
+ // initialization; window/grouping-only builtins (current_user, lag, lead,
...) are Unevaluable
+ // outside their normal context; and a type mismatch the analyzer's
implicit-cast pass would
+ // normally have caught still fails checkInputDataTypes - checked both on
the raw lookup result
+ // (its own declared input-type contract, e.g. split_part's, is otherwise
discarded once
+ // unwrapped) and again after unwrapping and widening (e.g. nvl(ts, 0) only
becomes checkable
+ // once it's the Coalesce(ts, 0) the hardcoded coalesce(ts, 0) case would
already have widened).
+ // Anything in one of those categories is treated as still-unresolved so it
falls through to the
+ // existing rejection path instead of silently dropping every row.
+ private def resolveViaFunctionRegistry(unresolvedFunc: UnresolvedFunction,
sparkSession: SparkSession): Expression = {
+ Try {
+ // Filter expressions only ever call plain builtins. FunctionRegistry
registers builtins
+ // with no database, so a db-qualified or 3+ part name (db.func,
catalog.db.func) can only
+ // be resolved by guessing which part is the real function name - that
risks matching an
+ // unrelated same-named function, so those are left unresolved instead.
+ val resolved = unresolvedFunc.nameParts match {
+ case Seq(funcName) =>
+
sparkSession.sessionState.functionRegistry.lookupFunction(FunctionIdentifier(funcName),
unresolvedFunc.arguments)
+ case _ => unresolvedFunc
+ }
+ // lookupFunction alone skips the analyzer's own implicit-cast rule, so
a wrapper declaring
+ // a real input-type contract (nvl needing matching operand types,
split_part needing
+ // string/string/int, ...) sees its raw, uncast arguments here. Casting
via that same rule
+ // before checking the contract lets a fixable mismatch (nvl(ts, 0), a
Long/Int pair) widen
+ // the way coalesce(ts, 0) already does, while a genuine mismatch
(split_part's delimiter
+ // passed as Int, which nothing implicit-casts to String) still fails as
it should.
+ val castedResolved = applyImplicitCasts(resolved)
Review Comment:
Addressed at 467472bdeed7. `applyImplicitCasts` now folds
`FunctionArgumentConversion`, `ConcatCoercion`, `IfCoercion` and
`ImplicitTypeCasts` (the ANSI engine under ANSI) over the lookup result.
Verified base-vs-head on the `show_cleans` OUTPUT_TYPE schema, Spark 3.5.5.
Each of these was `Unsupported functions: ...` at 97c30d35dfd7 and now resolves
with the same row count `df.filter` returns: `concat(version, 'x') = '2x'` (2),
`concat(time_taken_in_millis, 'x') = '150x'` (1), `if(total_files_deleted > 0,
time_taken_in_millis, 0) = 150` (1), `greatest(time_taken_in_millis, 0) > 100`
(1), `least(version, time_taken_in_millis) = 2` (2). Genuine mismatches still
reject: `concat(action, array(1))`, `split_part(action, array(1), 1)`.
The file also compiles clean against 3.3.4 / 3.4.3 / 3.5.5 / 4.0.2 / 4.1.1 /
4.2.0 (`TypeCoercionBase` and all three named rules exist on every one), and
`TestHoodieProcedureFilterUtils` + `TestShowCleansProcedures` are 44/44 green
locally on spark3.5 / JDK 11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
