Hey Josep et al, I've created a ticket regarding this. https://issues.apache.org/jira/browse/KAFKA-16237
All the best, Matthias Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat <josep.p...@aiven.io.invalid>: > > Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when > your accounts are created and we'll properly set them up so you can create > and assign tickets to you. > > Best, > > On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <matthias.ber...@ttmzero.com> > wrote: > > > Thanks Josep, I've applied for a JIRA account and addressed your > > review comments. > > > > Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > <josep.p...@aiven.io.invalid>: > > > > > > Hi Matthias, > > > > > > I think for this particular case it would be worth creating a JIRA ticket > > > for this as it's a new "feature". > > > Regarding the change itself, I think we need to clarify how the release > > > process would work. Right now, the script `gradlewAll` is used (which > > > basically runs the build with Scala version 2.12 and 2.13). If I > > understand > > > your changes correctly, we would need to run the build 3 times: > > > - 1 with property scalaVersion 2.12 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > > > I think we should document this and discuss when to have this feature. > > If I > > > remember correctly from when I tried to update Kafka to Scala 3, the idea > > > was to push this to a Kafka 4.0 version because we didn't want to > > maintain > > > more than 2 Scala versions at the same time. I would encourage if not > > > having a KIP, at least open up a [DISCUSS] thread to clarify some of > > these > > > points. > > > > > > I'll add some feedback on the PR itself regarding the changes. > > > > > > Best, > > > > > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > > matthias.ber...@ttmzero.com> > > > wrote: > > > > > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > > > > > Thank you for your encouraging responses regarding a Scala 3 port of > > > > Kafka-Streams-Scala, and apologies for the late response from my side. > > > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > > > > retaining support for 2.13 and 2.12). Almost no changes to the code > > > > were required and the tests also pass. Please take a look and let me > > > > know what you think :-) > > > > https://github.com/apache/kafka/pull/15338 > > > > > > > > All the best > > > > Matthias > > > > > > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > > > <josep.p...@aiven.io.invalid>: > > > > > > > > > > Hi, > > > > > > > > > > For reference, prior work on this: > > > > > https://github.com/apache/kafka/pull/11350 > > > > > https://github.com/apache/kafka/pull/11432 > > > > > > > > > > Best, > > > > > > > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbruts...@confluent.io > > > > .invalid> > > > > > wrote: > > > > > > > > > > > Hi Matthiases, > > > > > > > > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that > > add > > > > > > Scala 3 support. However, as Matthias S. said, it has to be driven > > by > > > > > > people who use Scala day-to-day, since I believe most Kafka Streams > > > > > > committers are working with Java. > > > > > > > > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a > > large > > > > > > undertaking, so option 1 is the first thing we should explore. > > > > > > > > > > > > I don't have any experience with Scala 3 migration topics, but on > > the > > > > > > Scala website it says > > > > > > > The first piece of good news is that the Scala 3 compiler is > > able to > > > > > > read the Scala 2.13 Pickle format and thus it can type check code > > that > > > > > > depends on modules or libraries compiled with Scala 2.13. > > > > > > > One notable example is the Scala 2.13 library. We have indeed > > decided > > > > > > that the Scala 2.13 library is the official standard library for > > Scala > > > > 3. > > > > > > So wouldn't that mean that we are safe in terms of standard library > > > > > > upgrades if we use core_2.13 in the tests? > > > > > > > > > > > > Cheers, > > > > > > Lucas > > > > > > > > > > > > > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> > > > > wrote: > > > > > > > > > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems > > to > > > > be an > > > > > > > important feature for Kafka Streams and I am generally in favor > > of > > > > your > > > > > > > proposal to add Scala 3 support. However, I am personally no > > Scala > > > > > > > person and it sounds like quite some overhead. > > > > > > > > > > > > > > If you are willing to drive and own this initiative happy to > > support > > > > you > > > > > > > to the extend I can. > > > > > > > > > > > > > > About the concrete proposal: my understanding is that :core will > > move > > > > > > > off Scala long-term (not 100% sure what the timeline is, but new > > > > modules > > > > > > > are written in Java only). Thus, down the road the compatibility > > > > issue > > > > > > > would go away naturally, but it's unclear when. > > > > > > > > > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it > > seems we > > > > > > > could add support for Scala 3 now, taking a risk that it might > > break > > > > in > > > > > > > the future assume that the migration off Scala from core is not > > fast > > > > > > enough. > > > > > > > > > > > > > > For proposal (2), I don't think that it would be easily possible > > for > > > > > > > unit/integration tests. We could fall back to system tests > > though, > > > > but > > > > > > > they would be much more heavy weight of course. > > > > > > > > > > > > > > Might be good to hear from others. We might actually also want > > to do > > > > a > > > > > > > KIP for this? > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > > > > > > Hey there, > > > > > > > > > > > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala > > > > library. > > > > > > > > Currently, the build system is set up such that > > kafka-streams-scala > > > > > > > > and core (i. e. kafka itself) are compiled with the same Scala > > > > > > > > compiler versions. This is not an optimal situation because it > > > > means > > > > > > > > that a Scala 3 release of kafka-streams-scala cannot happen > > > > > > > > independently of kafka itself. I think this should be changed > > > > > > > > > > > > > > > > The production codebase of scala-streams-kafka actually > > compiles > > > > just > > > > > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. > > The > > > > > > > > problem is with the tests. These use the `EmbeddedKafkaCluster` > > > > class, > > > > > > > > which means that kafka is pulled into the classpath, > > potentially > > > > > > > > leading to binary compatibility issues. > > > > > > > > I can see several approaches to fixing this: > > > > > > > > > > > > > > > > 1. Run the kafka-streams-scala tests using the compatible > > version > > > > of > > > > > > > > :core if one is available. Currently, this means that > > everything > > > > can > > > > > > > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > > > > > > > kafka-streams-scala_2.13 using core_2.13 and > > kafka-streams-scala_3 > > > > > > > > using core_2.13, as these should be compatible), but when a new > > > > > > > > scala-library version is released that is no longer compatible > > with > > > > > > > > 2.13, we won't be able to test that. > > > > > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, > > instead > > > > > > > > running the test cluster in a separate JVM or perhaps even a > > > > > > > > container. > > > > > > > > > > > > > > > > I'd be willing to get my hands dirty working on this, but > > before I > > > > > > > > start I'd like to get some feedback from the Kafka team > > regarding > > > > the > > > > > > > > approaches outlined above. > > > > > > > > > > > > > > > > All the best > > > > > > > > Matthias Berndt > > > > > > > > > > > KJosep Prat > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io | > > > > > +491715557497 | aiven.io > > > > > Aiven Deutschland GmbH > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Open Source Engineering Director, *Aiven* > > > josep.p...@aiven.io | +491715557497 > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud> > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B
