[
https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258288#comment-17258288
]
Stephan Ewen edited comment on FLINK-13414 at 1/4/21, 3:45 PM:
---------------------------------------------------------------
Thanks a lot everyone for this effort. Improving Scala support and evolution
would be really great.
In general, taking ownership of topics that have not great ownership at the
moment is invaluable to the community and one of the best ways to get
recognized fast.
I would like to throw in a completely different idea about handling Scala in
Flink: *Separating the "Framework Scala" from the "API Scala"*
What does that mean, concretely: It would mean that Flink runs two versions of
Scala in parallel in different class loaders: The Framework and the
API/Applications.
The Flink runtime and SQL engine can have a dependency on a specific Scala
version (say 2.12.x) while the API can depend on a completely different Scala
version. We would not have Scala Version suffixes any more on any module except
for the Scala API modules.
Flink does something similar with other dependencies already: The application
code (DataStream API program) runs in a dedicated classloader with inverted
classloading (child first). Because of that, users can use different versions
of dependencies in their programs than Flink has in the classpath. However,
this approach does only work for dependencies that do leak through the API
surface. Otherwise inverted classloading cannot be used, it leads to class
duplication and the infamous "X cannot be cast to X" exceptions.
So one way to fix this would be to make sure that the API surface is Java only,
and remove Scala from the "parent first patterns" in Flink's classloading, see
https://ci.apache.org/projects/flink/flink-docs-stable/ops/debugging/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
The biggest obstacle to that is, to my understanding, the Serialization stack.
If we can clean this up so that the surface does not show any Scala any more,
and all Scala dependencies and classes (as used by the APIs) come only from the
"flink-streaming-scala" module, then we have solved most of this.
In the future, supporting a new Scala version in the APIs means we need to look
at the "flink-streaming-scala" module, and nothing else. It would be much
simpler.
was (Author: stephanewen):
Thanks a lot everyone for this effort. Improving Scala support and evolution
would be really great.
In general, taking ownership of topics that have not great ownership at the
moment is invaluable to the community and one of the best ways to get
recognized fast.
I would like to throw in a completely different idea about handling Scala in
Flink: *Separating the "Framework Scala" from the "API Scala"*
What does that mean, concretely: It would mean that Flink runs two versions of
Scala in parallel in different class loaders: The Framework and the
API/Applications.
The Flink runtime and SQL engine can have a dependency on a specific Scala
version (say 2.12.x) while the API can depend on a completely different Scala
version. We would not have Scala Version suffixes any more on any module except
for the Scala API modules.
Flink does something similar with other dependencies already: The application
code (DataStream API program) runs in a dedicated classloader with inverted
classloading (child first). Because of that, users can use different versions
of dependencies in their programs than Flink has in the classpath. However,
this approach does only work for dependencies that do leak through the API
surface. Otherwise inverted classloading cannot be used, it leads to class
duplication and the infamous "X cannot be cast to X" exceptions.
So one way to fix this would be to make sure that the API surface is Java only,
and remove Scala from the "parent first patterns" in Flink's classloading, see
https://ci.apache.org/projects/flink/flink-docs-stable/ops/debugging/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order
The biggest obstacle to that is, to my understanding, the Serialization stack.
If we can clean this up so that the surface does not show any Scala any more,
and all Scala dependencies and classes (as used by the APIs) come only from the
"flink-streaming-scala" module, then we have solved most of this.
> Add support for Scala 2.13
> --------------------------
>
> Key: FLINK-13414
> URL: https://issues.apache.org/jira/browse/FLINK-13414
> Project: Flink
> Issue Type: New Feature
> Components: API / Scala
> Reporter: Chaoran Yu
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
