anshulsingh-py commented on code in PR #19853:
URL: https://github.com/apache/hudi/pull/19853#discussion_r4013633385
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedureFilterUtils.scala:
##########
@@ -389,6 +418,155 @@ object HoodieProcedureFilterUtils {
}
}
+ // Resolves a function not covered by the hardcoded table above via Spark's
own FunctionRegistry,
+ // then checks the result is actually usable outside a real query plan -
both steps a plain
+ // lookupFunction call skips or can't tell on its own. Anything that isn't
falls through to the
+ // existing rejection path instead of letting eval() throw silently.
+ private def resolveViaFunctionRegistry(unresolvedFunc: UnresolvedFunction,
sparkSession: SparkSession): Expression = {
+ Try {
+ val castedResolved =
applySparkAnalyzerCoercionRules(lookupBuiltin(unresolvedFunc, sparkSession))
+ // Checked here, on the raw wrapper, before unwrapping: a
RuntimeReplaceable wrapper's own
+ // declared input-type contract (nvl needing matching operand types,
split_part needing
+ // string/string/int) is otherwise discarded once unwrapped to a form
with a weaker or
+ // absent contract of its own.
+ if (!castedResolved.checkInputDataTypes().isSuccess) {
+ unresolvedFunc
+ } else {
+ val finalized = unwrapAndWiden(castedResolved)
+ if (isUsableOutsideQueryPlan(finalized)) finalized else unresolvedFunc
+ }
+ }.getOrElse(unresolvedFunc)
Review Comment:
Added the debug log - this catch-all is exactly what made the Spark 4.2
FunctionIdentifier breakage undiagnosable earlier in this PR, so that part is a
real improvement.
Kept it as a catch-all rather than narrowing to specific analysis failures,
though: that 4.2 breakage surfaced as a java.lang.AssertionError from
FunctionRegistry's internal assert, not an AnalysisException, so the narrower
filter would have let a version mismatch escape as a user-facing crash instead
of reaching the rejection path. What a given version throws for an unknown or
wrong-arity name isn't a stable contract, so the fallback stays broad and the
log carries the diagnosis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
