Dustin Smith created SPARK-60846:
------------------------------------
Summary: Accept Decimal from Spark 4.3.0 clients in Scala UDF
payloads
Key: SPARK-60846
URL: https://issues.apache.org/jira/browse/SPARK-60846
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Dustin Smith
{{UdfSerialization.auditedTransitions}} lets a Spark Connect server read Scala
UDF payloads whose {{org.apache.spark.sql.types}} classes have a different
computed {{serialVersionUID}} than the server's build, for audited transitions
only. The table was audited against the published {{spark-sql-api_2.13}} jars
of 4.0.0 to 4.0.4, 4.1.0 to 4.1.3 and 4.2.0.
Spark 4.3.0 shipped {{org.apache.spark.sql.types.Decimal}} with the computed
{{serialVersionUID}} {{4401554505066564464}}, which matches neither the earlier
releases ({{1715621871942419369}}) nor master. Master has no transition from
it, so a Scala UDF payload that carries a {{Decimal}} (for example a UDF
closure that captures a {{Decimal}}) from a 4.3.0 client is rejected with
{{InvalidClassException}}.
Proposed change: audit the classes in the table against the published 4.0.5,
4.1.4, 4.2.1 and 4.3.0 jars and add the missing transitions, backed by a
recorded 4.3.0 class descriptor in the {{UdfSerializationSuite}} test
resources. Only {{Decimal}} in 4.3.0 needs a new entry: its fields and
serialization hooks are identical to the already-audited releases, and every
other audited class in these releases has a {{serialVersionUID}} that is
already covered.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
