Hi Matthias, Yes, it's just a matter of adding the [DISCUSS] prefix in the subject. By the way, I didn't say this won't need a KIP, just that I won't be pushing for it, but other maintainers might think it's needed.
For the discuss thread, you should write down what changes in the build and what steps would be needed to create the artifacts. Best, Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Feb 9, 2024, 19:55 Matthias Berndt <matthias.ber...@ttmzero.com> wrote: > Hey Josep, > > I'm glad you agree that a KIP is not needed here, and I agree with you that > how to publish these artifacts should be discussed with the Kafka team. In > fact, this is what I created this thread for 😁 This is my first time > contributing to Kafka, so I'm going to have to ask what a DISCUSS thread > is. Is it just a mailing list thread with a subject that starts with > [DISCUSS], or is there more behind it? > > Best regards, > Matthias > > Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat > <josep.p...@aiven.io.invalid>: > > > Hi Matthias, > > It's not adding a new functionality but it's changing the way to generate > > artifacts. In the end we are talking about generating a new binary. > > > > I could live with not having a KIP, but a DISCUSS thread I think it's > > necessary. This signals the community members and maintainers that their > > input is needed. > > > > I could help you with writing the KIP if you want. > > > > Best, > > > > Josep Prat > > Open Source Engineering Director, aivenjosep.p...@aiven.io | > > +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Fri, Feb 9, 2024, 18:02 Matthias Berndt <matthias.ber...@ttmzero.com> > > wrote: > > > > > Hi Matthias, Hi Josep, > > > > > > I'm afraid I can't do the KIP thing as the signup process for Apache > > > Confluence requires sending me a password reset link via E-Mail and > said > > > E-Mail doesn't seem to reach me for some reason. I've contacted the > > Apache > > > infrastructure team but haven't yet heard back from them. > > > That said, I'd like to push back on the notion that a KIP is really > > > necessary for this change. It's certainly not a “major new feature” as > it > > > adds zero extra functionality, and it doesn't affect binary > compatibility > > > either as all the currently supported Scala versions are still > supported. > > > This looks like a routine upgrade to me. Please, let's try to keep the > > > administrative overhead to the required minimum, shall we? > > > Thanks btw for merging Github PR #15239 (removal of > > scala-collection-compat > > > dependency from the 2.13 artifact). That will already improve life for > > > Scala 3 users. > > > > > > All the best, > > > Matthias > > > > > > > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > > > mj...@apache.org > > > >: > > > > > > > Josep, > > > > > > > > thanks for helping with this. I was also thinking if we might need a > > KIP > > > > for this change. Since you had the same though, I would say, yes, > let's > > > > do a KIP. > > > > > > > > @Matthias: can you prepare a KIP? You can read up on the details on > the > > > > wiki page: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > If you have any questions about the process, please let us know. > > > > > > > > Thanks for pushing this forward! > > > > > > > > > > > > -Matthias > > > > > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > > > Hey Josep et al, > > > > > > > > > > I've created a ticket regarding this. > > > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > > > > > All the best, > > > > > Matthias > > > > > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > > > <josep.p...@aiven.io.invalid>: > > > > >> > > > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us > > know > > > > when > > > > >> your accounts are created and we'll properly set them up so you > can > > > > create > > > > >> and assign tickets to you. > > > > >> > > > > >> Best, > > > > >> > > > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > > > > matthias.ber...@ttmzero.com> > > > > >> wrote: > > > > >> > > > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > > > >>> review comments. > > > > >>> > > > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > > > >>> <josep.p...@aiven.io.invalid>: > > > > >>>> > > > > >>>> Hi Matthias, > > > > >>>> > > > > >>>> I think for this particular case it would be worth creating a > JIRA > > > > ticket > > > > >>>> for this as it's a new "feature". > > > > >>>> Regarding the change itself, I think we need to clarify how the > > > > release > > > > >>>> process would work. Right now, the script `gradlewAll` is used > > > (which > > > > >>>> basically runs the build with Scala version 2.12 and 2.13). If I > > > > >>> understand > > > > >>>> your changes correctly, we would need to run the build 3 times: > > > > >>>> - 1 with property scalaVersion 2.12 > > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > >>>> > > > > >>>> I think we should document this and discuss when to have this > > > feature. > > > > >>> If I > > > > >>>> remember correctly from when I tried to update Kafka to Scala 3, > > the > > > > idea > > > > >>>> was to push this to a Kafka 4.0 version because we didn't want > to > > > > >>> maintain > > > > >>>> more than 2 Scala versions at the same time. I would encourage > if > > > not > > > > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify > some > > of > > > > >>> these > > > > >>>> points. > > > > >>>> > > > > >>>> I'll add some feedback on the PR itself regarding the changes. > > > > >>>> > > > > >>>> Best, > > > > >>>> > > > > >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > > > > >>> matthias.ber...@ttmzero.com> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> Hi Matthias J., Hi Lucas, Hi Josep, > > > > >>>>> > > > > >>>>> Thank you for your encouraging responses regarding a Scala 3 > port > > > of > > > > >>>>> Kafka-Streams-Scala, and apologies for the late response from > my > > > > side. > > > > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3 > > > (while > > > > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the > > code > > > > >>>>> were required and the tests also pass. Please take a look and > let > > > me > > > > >>>>> know what you think :-) > > > > >>>>> https://github.com/apache/kafka/pull/15338 > > > > >>>>> > > > > >>>>> All the best > > > > >>>>> Matthias > > > > >>>>> > > > > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > > > >>>>> <josep.p...@aiven.io.invalid>: > > > > >>>>>> > > > > >>>>>> Hi, > > > > >>>>>> > > > > >>>>>> For reference, prior work on this: > > > > >>>>>> https://github.com/apache/kafka/pull/11350 > > > > >>>>>> https://github.com/apache/kafka/pull/11432 > > > > >>>>>> > > > > >>>>>> Best, > > > > >>>>>> > > > > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy < > > lbruts...@confluent.io > > > > >>>>> .invalid> > > > > >>>>>> wrote: > > > > >>>>>> > > > > >>>>>>> Hi Matthiases, > > > > >>>>>>> > > > > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes > > > that > > > > >>> add > > > > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be > > > driven > > > > >>> by > > > > >>>>>>> people who use Scala day-to-day, since I believe most Kafka > > > Streams > > > > >>>>>>> committers are working with Java. > > > > >>>>>>> > > > > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems > like > > a > > > > >>> large > > > > >>>>>>> undertaking, so option 1 is the first thing we should > explore. > > > > >>>>>>> > > > > >>>>>>> I don't have any experience with Scala 3 migration topics, > but > > on > > > > >>> the > > > > >>>>>>> Scala website it says > > > > >>>>>>>> The first piece of good news is that the Scala 3 compiler is > > > > >>> able to > > > > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check > > code > > > > >>> that > > > > >>>>>>> depends on modules or libraries compiled with Scala 2.13. > > > > >>>>>>>> One notable example is the Scala 2.13 library. We have > indeed > > > > >>> decided > > > > >>>>>>> that the Scala 2.13 library is the official standard library > > for > > > > >>> Scala > > > > >>>>> 3. > > > > >>>>>>> So wouldn't that mean that we are safe in terms of standard > > > library > > > > >>>>>>> upgrades if we use core_2.13 in the tests? > > > > >>>>>>> > > > > >>>>>>> Cheers, > > > > >>>>>>> Lucas > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax < > > > mj...@apache.org> > > > > >>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module > > seems > > > > >>> to > > > > >>>>> be an > > > > >>>>>>>> important feature for Kafka Streams and I am generally in > > favor > > > > >>> of > > > > >>>>> your > > > > >>>>>>>> proposal to add Scala 3 support. However, I am personally no > > > > >>> Scala > > > > >>>>>>>> person and it sounds like quite some overhead. > > > > >>>>>>>> > > > > >>>>>>>> If you are willing to drive and own this initiative happy to > > > > >>> support > > > > >>>>> you > > > > >>>>>>>> to the extend I can. > > > > >>>>>>>> > > > > >>>>>>>> About the concrete proposal: my understanding is that :core > > will > > > > >>> move > > > > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but > > new > > > > >>>>> modules > > > > >>>>>>>> are written in Java only). Thus, down the road the > > compatibility > > > > >>>>> issue > > > > >>>>>>>> would go away naturally, but it's unclear when. > > > > >>>>>>>> > > > > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it > > > > >>> seems we > > > > >>>>>>>> could add support for Scala 3 now, taking a risk that it > might > > > > >>> break > > > > >>>>> in > > > > >>>>>>>> the future assume that the migration off Scala from core is > > not > > > > >>> fast > > > > >>>>>>> enough. > > > > >>>>>>>> > > > > >>>>>>>> For proposal (2), I don't think that it would be easily > > possible > > > > >>> for > > > > >>>>>>>> unit/integration tests. We could fall back to system tests > > > > >>> though, > > > > >>>>> but > > > > >>>>>>>> they would be much more heavy weight of course. > > > > >>>>>>>> > > > > >>>>>>>> Might be good to hear from others. We might actually also > want > > > > >>> to do > > > > >>>>> a > > > > >>>>>>>> KIP for this? > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -Matthias > > > > >>>>>>>> > > > > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > > >>>>>>>>> Hey there, > > > > >>>>>>>>> > > > > >>>>>>>>> I'd like to discuss a Scala 3 port of the > kafka-streams-scala > > > > >>>>> library. > > > > >>>>>>>>> Currently, the build system is set up such that > > > > >>> kafka-streams-scala > > > > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same > > Scala > > > > >>>>>>>>> compiler versions. This is not an optimal situation because > > it > > > > >>>>> means > > > > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen > > > > >>>>>>>>> independently of kafka itself. I think this should be > changed > > > > >>>>>>>>> > > > > >>>>>>>>> The production codebase of scala-streams-kafka actually > > > > >>> compiles > > > > >>>>> just > > > > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax > changes. > > > > >>> The > > > > >>>>>>>>> problem is with the tests. These use the > > `EmbeddedKafkaCluster` > > > > >>>>> class, > > > > >>>>>>>>> which means that kafka is pulled into the classpath, > > > > >>> potentially > > > > >>>>>>>>> leading to binary compatibility issues. > > > > >>>>>>>>> I can see several approaches to fixing this: > > > > >>>>>>>>> > > > > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible > > > > >>> version > > > > >>>>> of > > > > >>>>>>>>> :core if one is available. Currently, this means that > > > > >>> everything > > > > >>>>> can > > > > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12, > > > > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and > > > > >>> kafka-streams-scala_3 > > > > >>>>>>>>> using core_2.13, as these should be compatible), but when a > > new > > > > >>>>>>>>> scala-library version is released that is no longer > > compatible > > > > >>> with > > > > >>>>>>>>> 2.13, we won't be able to test that. > > > > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster, > > > > >>> instead > > > > >>>>>>>>> running the test cluster in a separate JVM or perhaps even > a > > > > >>>>>>>>> container. > > > > >>>>>>>>> > > > > >>>>>>>>> I'd be willing to get my hands dirty working on this, but > > > > >>> before I > > > > >>>>>>>>> start I'd like to get some feedback from the Kafka team > > > > >>> regarding > > > > >>>>> the > > > > >>>>>>>>> approaches outlined above. > > > > >>>>>>>>> > > > > >>>>>>>>> All the best > > > > >>>>>>>>> Matthias Berndt > > > > >>>>>>> > > > > >>>>>> KJosep Prat > > > > >>>>>> Open Source Engineering Director, aivenjosep.p...@aiven.io > | > > > > >>>>>> +491715557497 | aiven.io > > > > >>>>>> Aiven Deutschland GmbH > > > > >>>>>> Alexanderufer 3-7, 10117 Berlin > > > > >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B > > > > >>>>> > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> [image: Aiven] <https://www.aiven.io> > > > > >>>> > > > > >>>> *Josep Prat* > > > > >>>> Open Source Engineering Director, *Aiven* > > > > >>>> josep.p...@aiven.io | +491715557497 > > > > >>>> aiven.io <https://www.aiven.io> | < > > > > >>> https://www.facebook.com/aivencloud> > > > > >>>> <https://www.linkedin.com/company/aiven/> < > > > > >>> https://twitter.com/aiven_io> > > > > >>>> *Aiven Deutschland GmbH* > > > > >>>> Alexanderufer 3-7, 10117 Berlin > > > > >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > >>>> Amtsgericht Charlottenburg, HRB 209739 B > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> [image: Aiven] <https://www.aiven.io> > > > > >> > > > > >> *Josep Prat* > > > > >> Open Source Engineering Director, *Aiven* > > > > >> josep.p...@aiven.io | +491715557497 > > > > >> aiven.io <https://www.aiven.io> | < > > > > https://www.facebook.com/aivencloud> > > > > >> <https://www.linkedin.com/company/aiven/> < > > > > https://twitter.com/aiven_io> > > > > >> *Aiven Deutschland GmbH* > > > > >> Alexanderufer 3-7, 10117 Berlin > > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > >> Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > >
