DB Tsai created SPARK-60247:
-------------------------------
Summary: Spark Connect typed UDF query context points to
UserDefinedFunction.apply instead of user code
Key: SPARK-60247
URL: https://issues.apache.org/jira/browse/SPARK-60247
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: DB Tsai
Typed Dataset APIs in the Spark Connect client (groupByKey, filter, reduce,
etc.)
encode the user function as a scalar Scala UDF applied to col("*"), e.g.
udf.apply(col("*")). The DataFrame query context origin is captured in
Column.apply via CurrentOrigin.withOrigin, whose immediate caller is
org.apache.spark.sql.expressions.UserDefinedFunction.apply
(UserDefinedFunction.scala:69).
CurrentOrigin.withOrigin filters Spark frames using sparkCodePattern in
sql/api .../catalyst/trees/origin.scala, which matches
org.apache.spark.sql.[classic|connect].{functions,Column,Dataset,...} but NOT
org.apache.spark.sql.expressions.UserDefinedFunction (nor typed-API
intermediaries such as KeyValueGroupedDatasetImpl). With the default
spark.sql.stackTracesInDataFrameContext=1, the captured "non-Spark" frame is
therefore the internal UserDefinedFunction.apply rather than the user's
class/line.
Example resulting context for a failed groupByKey UDF unpack:
== DataFrame ==
"apply" was called from
org.apache.spark.sql.expressions.UserDefinedFunction.apply(UserDefinedFunction.scala:69)
Expected: the context should point to the user's application code.
Note the fix is more than whitelisting UserDefinedFunction: the next frame
(KeyValueGroupedDatasetImpl.apply, etc.) is also unmatched, so a robust fix
likely captures origin around the public typed-API entry points or extends the
frame filter across the whole typed-UDF construction chain. A regression test
should use a real client-generated expression and corrupt only its serialized
payload.
Follow-up to SPARK-60226 (server-side operation labels and origin preservation
in the Connect planner); the server-side handling there is correct, but the
client-supplied origin can point to Spark internals.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
