[
https://issues.apache.org/jira/browse/SPARK-60783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anil K updated SPARK-60783:
---------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> A V2 ScalarFunction that inherits produceResult or invoke from a base class
> fails with SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60783
> URL: https://issues.apache.org/jira/browse/SPARK-60783
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Anil K
> Priority: Major
>
> {{V2ExpressionUtils.resolveScalarFunction}} finds a {{ScalarFunction}}'s
> magic {{invoke}} method or its {{produceResult}} with {{findMethod}}, which
> calls {{getDeclaredMethod}} on the function's own class. That does not see a
> method the class inherits from a superclass. So a function that implements
> {{produceResult}} or {{invoke}} in a base class is reported as not
> implementing it:
> {code:scala}
> abstract class StrLenBase extends ScalarFunction[Int] {
> override def produceResult(input: InternalRow): Int =
> input.getUTF8String(0).numChars()
> }
> class InheritedStrLen extends StrLenBase {
> override def inputTypes(): Array[DataType] = Array(StringType)
> override def resultType(): DataType = IntegerType
> override def name(): String = "strlen_inherited"
> }
> {code}
> Measured on master (0aa4f9136bc), with the function in a catalog:
> * {{SELECT testcat.ns.inherited('abc')}} fails analysis with
> {{[SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED] ScalarFunction `strlen_inherited`
> not implements or overrides method 'produceResult(InternalRow)'}}.
> * The same function with {{produceResult}} declared on the class itself
> returns 3.
> {{resolveScalarFunction}} is used in three places, so the same function fails
> in each:
> * a SQL call of the function ({{FunctionResolution}});
> * the write path, which clusters rows by a partition transform
> ({{DistributionAndOrderingUtils}}, through
> {{TransformExpression.resolvedFunction}});
> * a storage-partitioned join that evaluates a partition transform (one-side
> shuffle, identity reduce).
> A fix could look the method up along the superclass chain (e.g. {{getMethod}}
> for public methods, or walking {{getSuperclass}} for {{getDeclaredMethod}}),
> stopping before the {{ScalarFunction}} interface, so that the interface's
> default {{produceResult}} is still not mistaken for an implementation.
> The existing TODO in {{resolveScalarFunction}} covers the opposite case: a
> Scala class that implements neither method still passes, since Scala adds a
> forwarder to the interface's default {{produceResult}} that
> {{getDeclaredMethod}} finds. Walking the superclass chain does not change
> that.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
