fangchenli commented on PR #57129:
URL: https://github.com/apache/spark/pull/57129#issuecomment-4986532828
Agreed on consensus + incremental. On the umbrella: SPARK-54150 ("Support
Scala 3.x") already exists and has 8 subtasks — both this PR (SPARK-58043) and
#57112 (SPARK-58029) are filed under it, along with SPARK-57549 (Scala 3
compile-time `AgnosticEncoder` derivation), SPARK-55152, SPARK-57548,
SPARK-57596 and SPARK-55157. It's typed as an Improvement rather than an
Umbrella; happy to convert it and start a dev@ thread before anything further
lands. I'd rather have the consensus first than keep sending one-off PRs.
On whether it's possible at all to support both Scala 2 and 3 from the same
source base — short answer: yes for the great majority of the code, with two
areas that would need version-specific sources. Some data rather than
speculation.
I compiled the full repo (29 Scala modules on the default profiles) with
`-Xsource:3`, which flags where 2.13 and 3 semantics diverge. Result: 267
warnings + 1 hard compile error, in 6 categories:
| feature | sites |
|---|---|
| `package-prefix-implicits` | 107 |
| `case-companion-function` | 90 * |
| `infer-override` | 47 |
| `case-apply-copy-access` | 20 |
| `no-infer-structural` | 3 |
| import `*` selector position | 1 (hard error) |
\* the sweep undercounts this one; a clean per-module recount gives 190,
which is why this PR touches 190 sites.
Everything except the 20 `case-apply-copy-access` sites is
behavior-preserving and compiles identically on 2.13 — they are plain source
cleanups, no `src/main/scala-2.13` / `src/main/scala-3` split required. That is
what these two PRs are. The `case-apply-copy-access` group is the one with real
bincompat consequences (it turns synthetic public `copy`/`apply` private, which
is the MiMa objection that stalled #50474) — I'd report and defer those rather
than touch them.
So syntax is not the hard part. The hard parts are:
1. **Reflection-based encoders.** `ScalaReflection`, `ExpressionEncoder`,
`Encoders`, `SQLImplicits` are built on `scala.reflect.runtime` + `TypeTag`
(~790 `TypeTag` references across `sql/` and `core/`). Scala 3 has no runtime
universe, so this needs real derivation work — which is exactly what
SPARK-57549 / SPARK-55152 / SPARK-57548 / SPARK-57596 contemplate. This is the
piece that decides "possible at all", and it's the one I'd want answered before
anyone commits to a plan.
2. **REPL.** `SparkILoop` and `Main` sit on `scala.tools.nsc`. Scala 3's
REPL is a different API; that module realistically needs a `scala-3` source dir
or a separate implementation.
Dependencies look mostly fine. Notably breeze 2.1.0 — the version we pin —
does cross-publish `breeze_3` and `breeze-macros_3`, so despite being
semi-retired (SPARK-51031) it is not a blocker.
@LuciferYang on the value question: the Connect clients do cover the "write
Scala 3 user code" case, and I agree they're the pragmatic answer for users
today. I don't think these two PRs are worth much on their own. The sweep was
mainly to find out whether the answer to @HyukjinKwon's question is yes or no,
and I believe it's "yes, modulo encoders and the REPL". If there's no appetite
for pursuing the umbrella, I'm happy to close both and put the sweep results on
SPARK-54150 instead so the data isn't lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
