Hi Matthiases, I know Scala 2 fairly well, so I'd be happy to review changes that add Scala 3 support. However, as Matthias S. said, it has to be driven by people who use Scala day-to-day, since I believe most Kafka Streams committers are working with Java.
Rewriting the tests to not use EmbeddedKafkaCluster seems like a large undertaking, so option 1 is the first thing we should explore. I don't have any experience with Scala 3 migration topics, but on the Scala website it says > The first piece of good news is that the Scala 3 compiler is able to read the > Scala 2.13 Pickle format and thus it can type check code that depends on > modules or libraries compiled with Scala 2.13. > One notable example is the Scala 2.13 library. We have indeed decided that > the Scala 2.13 library is the official standard library for Scala 3. So wouldn't that mean that we are safe in terms of standard library upgrades if we use core_2.13 in the tests? Cheers, Lucas On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org> wrote: > > Thanks for raising this. The `kafka-streams-scala` module seems to be an > important feature for Kafka Streams and I am generally in favor of your > proposal to add Scala 3 support. However, I am personally no Scala > person and it sounds like quite some overhead. > > If you are willing to drive and own this initiative happy to support you > to the extend I can. > > About the concrete proposal: my understanding is that :core will move > off Scala long-term (not 100% sure what the timeline is, but new modules > are written in Java only). Thus, down the road the compatibility issue > would go away naturally, but it's unclear when. > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > could add support for Scala 3 now, taking a risk that it might break in > the future assume that the migration off Scala from core is not fast enough. > > For proposal (2), I don't think that it would be easily possible for > unit/integration tests. We could fall back to system tests though, but > they would be much more heavy weight of course. > > Might be good to hear from others. We might actually also want to do a > KIP for this? > > > -Matthias > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > Hey there, > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library. > > Currently, the build system is set up such that kafka-streams-scala > > and core (i. e. kafka itself) are compiled with the same Scala > > compiler versions. This is not an optimal situation because it means > > that a Scala 3 release of kafka-streams-scala cannot happen > > independently of kafka itself. I think this should be changed > > > > The production codebase of scala-streams-kafka actually compiles just > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The > > problem is with the tests. These use the `EmbeddedKafkaCluster` class, > > which means that kafka is pulled into the classpath, potentially > > leading to binary compatibility issues. > > I can see several approaches to fixing this: > > > > 1. Run the kafka-streams-scala tests using the compatible version of > > :core if one is available. Currently, this means that everything can > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 > > using core_2.13, as these should be compatible), but when a new > > scala-library version is released that is no longer compatible with > > 2.13, we won't be able to test that. > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead > > running the test cluster in a separate JVM or perhaps even a > > container. > > > > I'd be willing to get my hands dirty working on this, but before I > > start I'd like to get some feedback from the Kafka team regarding the > > approaches outlined above. > > > > All the best > > Matthias Berndt
