fangchenli commented on PR #50474: URL: https://github.com/apache/spark/pull/50474#issuecomment-3892896760
@joan38 @sideeffffect A PR touching 300 files is almost impossible to review and merge. The only way forward is to break this PR into smaller ones and use [SPARK-54150](https://issues.apache.org/jira/browse/SPARK-54150) to track them. So the low-impact changes could get merged quickly. I'm not a Spark maintainer, it's just my personal opinion: For a project at this scale, stability is always prioritized over keeping everything up to date. Supporting Scala 3 is non-trivial and will definitely destabilize the codebase. Especially for changes like this, most end users probably use PySpark and couldn't care less about the Scala core. Without clear evidence of performance improvements and potential new features, it's common to place this kind of upgrade in the "nice to have" category. To actually unlock Scala 3, we need (at least): - Cross-build plumbing. I have two branches (no PRs yet) that enable native sbt build, which would make cross-build much easier. - Scala2-only dependency migration. https://github.com/apache/spark/pull/53943. I also forked chill and modernized it to support Scala 3 https://github.com/forge-labs-dev/chill. But personally, I think we should just move to [fory](https://github.com/apache/fory). It's way faster than Kryo. - Replace the reflection-based encoder with a Scala3 compile-time implementation. I have one implemented using the Mirror API in a separate repo. It's ready, but the cross-build must land first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
