I also noticed that we two replies in a separate thread on the User mailing list, which can be found at https://lists.apache.org/thread/m5ntl3cj81wg7frbfqg9v75c7hqnxtls.
I've included Clayton and David in this email, to at least centralize the conversation once more :) On Wed, Oct 5, 2022 at 11:36 AM Martijn Visser <martijnvis...@apache.org> wrote: > @Maciek > > I saw that I missed replying to your question: > > > Could you please remind what was the conclusion of discussion on > upgrading Scala to 2.12.15/16? > > https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I > couldn't find any follow-up vote? > > There is a vote thread, but that never got enough votes. See > https://lists.apache.org/thread/l93l5qqr5n2oty3r2bjsz3ks3tjf1655 > > > If it's acceptable to break binary compatibility by such an upgrade, > then upgrading to JDK17 before 2.0 will be doable? > > I'm not sure, because I don't think a discussion and vote has been made > yet if upgrading JDK17 can/will be done in a Flink 1.0 release or if it > requires a 2.0 release. It was mentioned in the original discussion thread > on dropping Java 8 support within 2/3 releases, but if I recall correctly > there was no discussion yet on when Java 17 support would be added [1]. > > Best regards, > > Martijn > > [1] https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx > > On Wed, Oct 5, 2022 at 6:58 AM Gaël Renoux <gael.ren...@datadome.co> > wrote: > >> > I'm curious what target Scala versions people are currently interested >> in. >> > I would've expected that everyone wants to migrate to Scala 3, for >> which several wrapper projects around Flink already exist >> >> The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not >> sure I would move my team to Scala 3 right now (I'm currently toying with >> it on a personal project). In addition, moving to Scala 3 is not completely >> free - you have to do some rewrites, and developers will need some >> adaptation time. Scala 2.13 is another thing entirely, we've wanted to >> migrate for a long while. >> >> On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ches...@apache.org> >> wrote: >> >>> > It's possible that for the sake of the Scala API, we would >>> occasionally require some changes in the Java API. As long as those changes >>> are not detrimental to Java users, they should be considered. >>> >>> That's exactly the model we're trying to get to. Don't fix >>> scala-specific issues with scala code, but rather on the Java side as much >>> as possible which could also benefit other JVM languages (e.g., Kotlin). >>> >>> > A question regarding the Flink wrapper: would it be possible to keep >>> it under the Flink project's umbrella? Or does it need to be a completely >>> separate structure? I'm not aware of the full organizational implications >>> of this, I'm afraid. >>> >>> Technically it can be under the Flink umbrella, but then Flink would >>> still be (at least for a while) be the bottleneck because we'd have to >>> review any changes coming in. >>> That would only improve once several new committers were added to take >>> care of this project. >>> (In the end you'd just split Flink and the Scala API _codebases_, but >>> achieve little else) >>> >>> > And if that is what it takes to move beyond Scala 2.12.7… This has >>> been a big pain point for us. >>> >>> I'm curious what target Scala versions people are currently interested >>> in. >>> I would've expected that everyone wants to migrate to Scala 3, for which >>> several wrapper projects around Flink already exist. >>> >>> On 05/10/2022 12:35, Gaël Renoux wrote: >>> >>> Hello everyone, >>> >>> I've already answered a bit on Twitter, I'll develop my thoughts a bit >>> here. For context, my company (DataDome) has a significant codebase on >>> Scala Flink (around 110K LOC), having been using it since 2017. I myself am >>> an enthusiastic Scala developer (I don't think I'd like moving back to >>> Java) >>> >>> Given that, I think separating the Scala support from Flink is actually >>> a good move long term. We would then have a full-Java Flink, and a separate >>> Scala wrapper. It would help a lot in solving the skills issue: Flink >>> maintainers would no longer need to be fluent in Scala, and maintainers of >>> the Scala wrapper would not need a deep knowledge of Flink's inner >>> workings, just the API would be sufficient. And if that is what it takes to >>> move beyond Scala 2.12.7… This has been a big pain point for us. >>> >>> I'm not too worried about finding contributors for the Scala wrapper. >>> Within my company, we have developed additional wrappers and extension >>> methods (for parts where we felt the Flink Scala API was insufficient), and >>> we've been looking at ways we could contribute back. What held us back was >>> our lack of knowledge of the full Flink environment (we're only using the >>> Scala Datastream API). I don't think we're the only ones in that situation. >>> One major point, though, is that Flink developers would not be completely >>> rid of us ;-) It's possible that for the sake of the Scala API, we would >>> occasionally require some changes in the Java API. As long as those changes >>> are not detrimental to Java users, they should be considered. >>> >>> A question regarding the Flink wrapper: would it be possible to keep it >>> under the Flink project's umbrella? Or does it need to be a completely >>> separate structure? I'm not aware of the full organizational implications >>> of this, I'm afraid. >>> >>> Finally, the hard part would be the migration to the new version. My >>> dream solution would be to have the existing Scala API be entirely >>> converted into a Scala wrapper over the Java API. That way, migration would >>> be pretty minimal: add a dependency, change the imports for the Scala API, >>> and we're done. However, even starting from the existing flink4s project, >>> that's still quite a lot of work. So, more realistically, I'd settle for at >>> least a partial implementation. We would have some broken code that we >>> could fix, but at the very least I'd like the basic DataStream functions >>> (process, uid, name…) to be available. >>> >>> Thanks for all the work that went into making Flink what it is! >>> >>> >>> Gaël Renoux - Lead R&D Engineer >>> E - gael.ren...@datadome.co >>> W - www.datadome.co >>> >>> >>> >>> On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <m...@touk.pl> wrote: >>> >>>> Hi Martin, >>>> >>>> Could you please remind what was the conclusion of discussion on >>>> upgrading Scala to 2.12.15/16? >>>> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I >>>> couldn't find any follow-up vote? >>>> >>>> If it's acceptable to break binary compatibility by such an upgrade, >>>> then upgrading to JDK17 before 2.0 will be doable? >>>> >>>> >>>> thanks, >>>> >>>> maciek >>>> >>>> >>>> On 04.10.2022 18:21, Martijn Visser wrote: >>>> >>>> Hi Yaroslav, >>>> >>>> Thanks for the feedback, that's much appreciated! Regarding Java 17 as >>>> a prerequisite, we would have to break compatibility already since Scala >>>> 2.12.7 doesn't compile on Java 17 [1]. >>>> >>>> Given that we can only remove Scala APIs with the next major Flink >>>> (2.0) version, would that still impact you a lot? I do imagine that if we >>>> get to a Flink 2.0 version there would be more breaking involved anyway. >>>> The biggest consequence of deprecating support for Scala in Flink 1.x would >>>> be that new APIs would only be available in Java, but since these don't >>>> exist yet there would be no refactoring involved. I can imagine that we >>>> might change something in an existing API, but that would have certain >>>> compatibility guarantees already (depending if it's >>>> Public/PublicEvolving/Experimental). If a change would happen there, I >>>> think it would be smaller refactoring. >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-25000 >>>> >>>> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko < >>>> yaros...@goldsky.com> wrote: >>>> >>>>> Hi Martijn, >>>>> >>>>> As a Scala user, this change would affect me a lot and I'm not looking >>>>> forward to rewriting my codebase, and it's not even a very large one :) >>>>> >>>>> I'd like to suggest supporting Java 17 as a prerequisite ( >>>>> https://issues.apache.org/jira/browse/FLINK-15736). Things like >>>>> switch expressions and records could simplify the migration quite a bit. >>>>> Would you consider adding it to the FLIP? >>>>> >>>>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <j...@ververica.com> wrote: >>>>> >>>>>> Hi Martijn, >>>>>> >>>>>> Thanks for bringing this up. It is generally a great idea, so +1. >>>>>> >>>>>> Since both scala extension projects mentioned in the FLIP are still >>>>>> very young and I don't think they will attract more scala developers as >>>>>> Flink could just because they are external projects. It will be a big >>>>>> issue >>>>>> for users who have to rewrite their large codebases. Those users should >>>>>> be >>>>>> aware of the effort from now on and would better not count on those scala >>>>>> extension projects and prepare their migration plan before Flink 2.0. >>>>>> >>>>>> Best regards, >>>>>> Jing >>>>>> >>>>>> >>>>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser < >>>>>> martijnvis...@apache.org> wrote: >>>>>> >>>>>>> Hi Marton, >>>>>>> >>>>>>> You're making a good point, I originally wanted to include already >>>>>>> the User mailing list to get their feedback but forgot to do so. I'll do >>>>>>> some more outreach via other channels as well. >>>>>>> >>>>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala >>>>>>> API support in a future version of Flink. Your feedback on this topic is >>>>>>> very much appreciated. >>>>>>> >>>>>>> Regarding the large Scala codebase for Flink, a potential >>>>>>> alternative could be to have a wrapper for all Java APIs that makes them >>>>>>> available as Scala APIs. However, this still requires Scala maintainers >>>>>>> and >>>>>>> I don't think that we currently have those in our community. The easiest >>>>>>> solution for them would be to use the Java APIs directly. Yes it would >>>>>>> involve work, but we won't actually be able to remove the Scala APIs >>>>>>> until >>>>>>> Flink 2.0 so there's still time for that :) >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Martijn >>>>>>> >>>>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi < >>>>>>> balassi.mar...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Martjin, >>>>>>>> >>>>>>>> Thanks for compiling the FLIP. I agree with the sentiment that >>>>>>>> Scala poses >>>>>>>> considerable maintenance overhead and key improvements (like 2.13 >>>>>>>> or 2.12.8 >>>>>>>> supports) are hanging stale. With that said before we make this >>>>>>>> move we >>>>>>>> should attempt to understand the userbase affected. >>>>>>>> A quick Slack and user mailing list search does return quite a bit >>>>>>>> of >>>>>>>> results for scala (admittedly a cursory look at them suggest that >>>>>>>> many of >>>>>>>> them have to do with missing features in Scala that exist in Java >>>>>>>> or Scala >>>>>>>> versions). I would love to see some polls on this topic, we could >>>>>>>> also use >>>>>>>> the Flink twitter handle to ask the community about this. >>>>>>>> >>>>>>>> I am aware of users having large existing Scala codebases for >>>>>>>> Flink. This >>>>>>>> move would pose a very large effort on them, as they would need to >>>>>>>> rewrite >>>>>>>> much of their existing code. What are the alternatives in your >>>>>>>> opinion, >>>>>>>> Martjin? >>>>>>>> >>>>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser < >>>>>>>> martijnvis...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>> > Hi everyone, >>>>>>>> > >>>>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate >>>>>>>> and remove >>>>>>>> > Scala API support. Please take a look at >>>>>>>> > >>>>>>>> > >>>>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support >>>>>>>> > and provide your feedback. >>>>>>>> > >>>>>>>> > Best regards, >>>>>>>> > >>>>>>>> > Martijn >>>>>>>> > https://twitter.com/MartijnVisser82 >>>>>>>> > https://github.com/MartijnVisser >>>>>>>> > >>>>>>>> >>>>>>> >>>
