[
https://issues.apache.org/jira/browse/SPARK-60247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60247:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect typed UDF query context points to UserDefinedFunction.apply
> instead of user code
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-60247
> URL: https://issues.apache.org/jira/browse/SPARK-60247
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
>
> Typed Dataset APIs in the Spark Connect client (groupByKey, filter, reduce,
> etc.)
> encode the user function as a scalar Scala UDF applied to col("*"), e.g.
> udf.apply(col("*")). The DataFrame query context origin is captured in
> Column.apply via CurrentOrigin.withOrigin, whose immediate caller is
> org.apache.spark.sql.expressions.UserDefinedFunction.apply
> (UserDefinedFunction.scala:69).
> CurrentOrigin.withOrigin filters Spark frames using sparkCodePattern in
> sql/api .../catalyst/trees/origin.scala, which matches
> org.apache.spark.sql.[classic|connect].{functions,Column,Dataset,...} but NOT
> org.apache.spark.sql.expressions.UserDefinedFunction (nor typed-API
> intermediaries such as KeyValueGroupedDatasetImpl). With the default
> spark.sql.stackTracesInDataFrameContext=1, the captured "non-Spark" frame is
> therefore the internal UserDefinedFunction.apply rather than the user's
> class/line.
> Example resulting context for a failed groupByKey UDF unpack:
> == DataFrame ==
> "apply" was called from
>
> org.apache.spark.sql.expressions.UserDefinedFunction.apply(UserDefinedFunction.scala:69)
> Expected: the context should point to the user's application code.
> Note the fix is more than whitelisting UserDefinedFunction: the next frame
> (KeyValueGroupedDatasetImpl.apply, etc.) is also unmatched, so a robust fix
> likely captures origin around the public typed-API entry points or extends the
> frame filter across the whole typed-UDF construction chain. A regression test
> should use a real client-generated expression and corrupt only its serialized
> payload.
> Follow-up to SPARK-60226 (server-side operation labels and origin preservation
> in the Connect planner); the server-side handling there is correct, but the
> client-supplied origin can point to Spark internals.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
