Fangchen Li created SPARK-57548:
-----------------------------------
Summary: Avoid eager scala.reflect.runtime.universe initialization
in ScalaReflection
Key: SPARK-57548
URL: https://issues.apache.org/jira/browse/SPARK-57548
Project: Spark
Issue Type: Sub-task
Components: Spark Core, SQL
Affects Versions: 4.3.0
Reporter: Fangchen Li
ScalaReflection (sql-api) eagerly initializes `val universe =
scala.reflect.runtime.universe` in its object initializer. On the Scala 3.8+
stdlib this crashes — `FatalError: class Array does not have a member apply` —
an open Scala regression that names Spark: scala/scala3#25896.
Because it's in the static initializer, merely touching ScalaReflection trips
it, and the generated ser/deser code does (via encodeFieldNameToIdentifier /
findConstructor) — so even a pre-built AgnosticEncoder can't run from Scala 3.
Proposed (sql-api, ~3 lines, behavior-preserving on 2.13): - `val universe` ->
`lazy val universe` - encodeFieldNameToIdentifier via
scala.reflect.NameTransformer.encode - findConstructor's scala-reflect fallback
-> Java reflection
Independent of, and smaller than, replacing encoderFor itself (separate
follow-up).
Reference (2-line patch + a Scala-3 round-trip through Spark's unmodified
codegen): https://github.com/bearing-research/ProtoCatalyst
(spark-reflection-patch/, docs/scala3-encoder/REPORT.md §3).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
