Re: Kafka-Streams-Scala for Scala 3
Hi Matthias, Yes, it's just a matter of adding the [DISCUSS] prefix in the subject. By the way, I didn't say this won't need a KIP, just that I won't be pushing for it, but other maintainers might think it's needed. For the discuss thread, you should write down what changes in the build and what steps would be needed to create the artifacts. Best, Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Feb 9, 2024, 19:55 Matthias Berndt wrote: > Hey Josep, > > I'm glad you agree that a KIP is not needed here, and I agree with you that > how to publish these artifacts should be discussed with the Kafka team. In > fact, this is what I created this thread for This is my first time > contributing to Kafka, so I'm going to have to ask what a DISCUSS thread > is. Is it just a mailing list thread with a subject that starts with > [DISCUSS], or is there more behind it? > > Best regards, > Matthias > > Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat > : > > > Hi Matthias, > > It's not adding a new functionality but it's changing the way to generate > > artifacts. In the end we are talking about generating a new binary. > > > > I could live with not having a KIP, but a DISCUSS thread I think it's > > necessary. This signals the community members and maintainers that their > > input is needed. > > > > I could help you with writing the KIP if you want. > > > > Best, > > > > Josep Prat > > Open Source Engineering Director, aivenjosep.p...@aiven.io | > > +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Fri, Feb 9, 2024, 18:02 Matthias Berndt > > wrote: > > > > > Hi Matthias, Hi Josep, > > > > > > I'm afraid I can't do the KIP thing as the signup process for Apache > > > Confluence requires sending me a password reset link via E-Mail and > said > > > E-Mail doesn't seem to reach me for some reason. I've contacted the > > Apache > > > infrastructure team but haven't yet heard back from them. > > > That said, I'd like to push back on the notion that a KIP is really > > > necessary for this change. It's certainly not a “major new feature” as > it > > > adds zero extra functionality, and it doesn't affect binary > compatibility > > > either as all the currently supported Scala versions are still > supported. > > > This looks like a routine upgrade to me. Please, let's try to keep the > > > administrative overhead to the required minimum, shall we? > > > Thanks btw for merging Github PR #15239 (removal of > > scala-collection-compat > > > dependency from the 2.13 artifact). That will already improve life for > > > Scala 3 users. > > > > > > All the best, > > > Matthias > > > > > > > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > > > mj...@apache.org > > > >: > > > > > > > Josep, > > > > > > > > thanks for helping with this. I was also thinking if we might need a > > KIP > > > > for this change. Since you had the same though, I would say, yes, > let's > > > > do a KIP. > > > > > > > > @Matthias: can you prepare a KIP? You can read up on the details on > the > > > > wiki page: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > If you have any questions about the process, please let us know. > > > > > > > > Thanks for pushing this forward! > > > > > > > > > > > > -Matthias > > > > > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > > > Hey Josep et al, > > > > > > > > > > I've created a ticket regarding this. > > > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > > > > > All the best, > > > > > Matthias > > > > > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > > > : > > > > >> > > > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us > > know > > > > when > > > > >> your accounts are created and we'll properly set them up so you > can > > > > create > > > > >> and assign tickets to you. > > > > >> > > > > >> Best, > > > > >> > > > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > > > > matthias.ber...@ttmzero.com> > > > > >> wrote: > > > > >> > > > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > > > >>> review comments. > > > > >>> > > > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > > > >>> : > > > > > > > > Hi Matthias, > > > > > > > > I think for this particular case it would be worth creating a > JIRA > > > > ticket > > > > for this as it's a new "feature". > > > > Regarding the change itself, I think we need to clarify how the > > > > release > > > > process would work. Right now, the script `gradlewAll` is used > > > (which > > > >
Re: Kafka-Streams-Scala for Scala 3
Hey Josep, I'm glad you agree that a KIP is not needed here, and I agree with you that how to publish these artifacts should be discussed with the Kafka team. In fact, this is what I created this thread for This is my first time contributing to Kafka, so I'm going to have to ask what a DISCUSS thread is. Is it just a mailing list thread with a subject that starts with [DISCUSS], or is there more behind it? Best regards, Matthias Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat : > Hi Matthias, > It's not adding a new functionality but it's changing the way to generate > artifacts. In the end we are talking about generating a new binary. > > I could live with not having a KIP, but a DISCUSS thread I think it's > necessary. This signals the community members and maintainers that their > input is needed. > > I could help you with writing the KIP if you want. > > Best, > > Josep Prat > Open Source Engineering Director, aivenjosep.p...@aiven.io | > +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Fri, Feb 9, 2024, 18:02 Matthias Berndt > wrote: > > > Hi Matthias, Hi Josep, > > > > I'm afraid I can't do the KIP thing as the signup process for Apache > > Confluence requires sending me a password reset link via E-Mail and said > > E-Mail doesn't seem to reach me for some reason. I've contacted the > Apache > > infrastructure team but haven't yet heard back from them. > > That said, I'd like to push back on the notion that a KIP is really > > necessary for this change. It's certainly not a “major new feature” as it > > adds zero extra functionality, and it doesn't affect binary compatibility > > either as all the currently supported Scala versions are still supported. > > This looks like a routine upgrade to me. Please, let's try to keep the > > administrative overhead to the required minimum, shall we? > > Thanks btw for merging Github PR #15239 (removal of > scala-collection-compat > > dependency from the 2.13 artifact). That will already improve life for > > Scala 3 users. > > > > All the best, > > Matthias > > > > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > > mj...@apache.org > > >: > > > > > Josep, > > > > > > thanks for helping with this. I was also thinking if we might need a > KIP > > > for this change. Since you had the same though, I would say, yes, let's > > > do a KIP. > > > > > > @Matthias: can you prepare a KIP? You can read up on the details on the > > > wiki page: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > If you have any questions about the process, please let us know. > > > > > > Thanks for pushing this forward! > > > > > > > > > -Matthias > > > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > > Hey Josep et al, > > > > > > > > I've created a ticket regarding this. > > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > > > All the best, > > > > Matthias > > > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > > : > > > >> > > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us > know > > > when > > > >> your accounts are created and we'll properly set them up so you can > > > create > > > >> and assign tickets to you. > > > >> > > > >> Best, > > > >> > > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > > > matthias.ber...@ttmzero.com> > > > >> wrote: > > > >> > > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > > >>> review comments. > > > >>> > > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > > >>> : > > > > > > Hi Matthias, > > > > > > I think for this particular case it would be worth creating a JIRA > > > ticket > > > for this as it's a new "feature". > > > Regarding the change itself, I think we need to clarify how the > > > release > > > process would work. Right now, the script `gradlewAll` is used > > (which > > > basically runs the build with Scala version 2.12 and 2.13). If I > > > >>> understand > > > your changes correctly, we would need to run the build 3 times: > > > - 1 with property scalaVersion 2.12 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > > > I think we should document this and discuss when to have this > > feature. > > > >>> If I > > > remember correctly from when I tried to update Kafka to Scala 3, > the > > > idea > > > was to push this to a Kafka 4.0 version because we didn't want to > > > >>> maintain > > > more than 2 Scala versions at the same time. I would encourage if > > not > > > having a KIP, at least open up a [DISCUSS] thread to clarify some > of > > > >>> these > > > points. > > > > > > I'll add some feedback on the PR itself
Re: Kafka-Streams-Scala for Scala 3
Hi Matthias, It's not adding a new functionality but it's changing the way to generate artifacts. In the end we are talking about generating a new binary. I could live with not having a KIP, but a DISCUSS thread I think it's necessary. This signals the community members and maintainers that their input is needed. I could help you with writing the KIP if you want. Best, Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Feb 9, 2024, 18:02 Matthias Berndt wrote: > Hi Matthias, Hi Josep, > > I'm afraid I can't do the KIP thing as the signup process for Apache > Confluence requires sending me a password reset link via E-Mail and said > E-Mail doesn't seem to reach me for some reason. I've contacted the Apache > infrastructure team but haven't yet heard back from them. > That said, I'd like to push back on the notion that a KIP is really > necessary for this change. It's certainly not a “major new feature” as it > adds zero extra functionality, and it doesn't affect binary compatibility > either as all the currently supported Scala versions are still supported. > This looks like a routine upgrade to me. Please, let's try to keep the > administrative overhead to the required minimum, shall we? > Thanks btw for merging Github PR #15239 (removal of scala-collection-compat > dependency from the 2.13 artifact). That will already improve life for > Scala 3 users. > > All the best, > Matthias > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > mj...@apache.org > >: > > > Josep, > > > > thanks for helping with this. I was also thinking if we might need a KIP > > for this change. Since you had the same though, I would say, yes, let's > > do a KIP. > > > > @Matthias: can you prepare a KIP? You can read up on the details on the > > wiki page: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > If you have any questions about the process, please let us know. > > > > Thanks for pushing this forward! > > > > > > -Matthias > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > Hey Josep et al, > > > > > > I've created a ticket regarding this. > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > All the best, > > > Matthias > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > : > > >> > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know > > when > > >> your accounts are created and we'll properly set them up so you can > > create > > >> and assign tickets to you. > > >> > > >> Best, > > >> > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > > matthias.ber...@ttmzero.com> > > >> wrote: > > >> > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > >>> review comments. > > >>> > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > >>> : > > > > Hi Matthias, > > > > I think for this particular case it would be worth creating a JIRA > > ticket > > for this as it's a new "feature". > > Regarding the change itself, I think we need to clarify how the > > release > > process would work. Right now, the script `gradlewAll` is used > (which > > basically runs the build with Scala version 2.12 and 2.13). If I > > >>> understand > > your changes correctly, we would need to run the build 3 times: > > - 1 with property scalaVersion 2.12 > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > I think we should document this and discuss when to have this > feature. > > >>> If I > > remember correctly from when I tried to update Kafka to Scala 3, the > > idea > > was to push this to a Kafka 4.0 version because we didn't want to > > >>> maintain > > more than 2 Scala versions at the same time. I would encourage if > not > > having a KIP, at least open up a [DISCUSS] thread to clarify some of > > >>> these > > points. > > > > I'll add some feedback on the PR itself regarding the changes. > > > > Best, > > > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > > >>> matthias.ber...@ttmzero.com> > > wrote: > > > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > > > Thank you for your encouraging responses regarding a Scala 3 port > of > > > Kafka-Streams-Scala, and apologies for the late response from my > > side. > > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 > (while > > > retaining support for 2.13 and 2.12). Almost no changes to the code > > > were required and the tests also pass. Please take a look and let > me > > > know what you think :-) > > > https://github.com/apache/kafka/pull/15338 > > > > > > All the best > >
Re: Kafka-Streams-Scala for Scala 3
Hi Matthias, Hi Josep, I'm afraid I can't do the KIP thing as the signup process for Apache Confluence requires sending me a password reset link via E-Mail and said E-Mail doesn't seem to reach me for some reason. I've contacted the Apache infrastructure team but haven't yet heard back from them. That said, I'd like to push back on the notion that a KIP is really necessary for this change. It's certainly not a “major new feature” as it adds zero extra functionality, and it doesn't affect binary compatibility either as all the currently supported Scala versions are still supported. This looks like a routine upgrade to me. Please, let's try to keep the administrative overhead to the required minimum, shall we? Thanks btw for merging Github PR #15239 (removal of scala-collection-compat dependency from the 2.13 artifact). That will already improve life for Scala 3 users. All the best, Matthias Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax : > Josep, > > thanks for helping with this. I was also thinking if we might need a KIP > for this change. Since you had the same though, I would say, yes, let's > do a KIP. > > @Matthias: can you prepare a KIP? You can read up on the details on the > wiki page: > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > If you have any questions about the process, please let us know. > > Thanks for pushing this forward! > > > -Matthias > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > Hey Josep et al, > > > > I've created a ticket regarding this. > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > All the best, > > Matthias > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > : > >> > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know > when > >> your accounts are created and we'll properly set them up so you can > create > >> and assign tickets to you. > >> > >> Best, > >> > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > matthias.ber...@ttmzero.com> > >> wrote: > >> > >>> Thanks Josep, I've applied for a JIRA account and addressed your > >>> review comments. > >>> > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > >>> : > > Hi Matthias, > > I think for this particular case it would be worth creating a JIRA > ticket > for this as it's a new "feature". > Regarding the change itself, I think we need to clarify how the > release > process would work. Right now, the script `gradlewAll` is used (which > basically runs the build with Scala version 2.12 and 2.13). If I > >>> understand > your changes correctly, we would need to run the build 3 times: > - 1 with property scalaVersion 2.12 > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > I think we should document this and discuss when to have this feature. > >>> If I > remember correctly from when I tried to update Kafka to Scala 3, the > idea > was to push this to a Kafka 4.0 version because we didn't want to > >>> maintain > more than 2 Scala versions at the same time. I would encourage if not > having a KIP, at least open up a [DISCUSS] thread to clarify some of > >>> these > points. > > I'll add some feedback on the PR itself regarding the changes. > > Best, > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > >>> matthias.ber...@ttmzero.com> > wrote: > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > Thank you for your encouraging responses regarding a Scala 3 port of > > Kafka-Streams-Scala, and apologies for the late response from my > side. > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > > retaining support for 2.13 and 2.12). Almost no changes to the code > > were required and the tests also pass. Please take a look and let me > > know what you think :-) > > https://github.com/apache/kafka/pull/15338 > > > > All the best > > Matthias > > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > : > >> > >> Hi, > >> > >> For reference, prior work on this: > >> https://github.com/apache/kafka/pull/11350 > >> https://github.com/apache/kafka/pull/11432 > >> > >> Best, > >> > >> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy > .invalid> > >> wrote: > >> > >>> Hi Matthiases, > >>> > >>> I know Scala 2 fairly well, so I'd be happy to review changes that > >>> add > >>> Scala 3 support. However, as Matthias S. said, it has to be driven > >>> by > >>> people who use Scala day-to-day, since I believe most Kafka Streams > >>> committers are working with Java. > >>> > >>> Rewriting the tests to not use EmbeddedKafkaCluster seems like a > >>> large > >>> undertaking, so option 1 is the first thing we should explore. > >>> > >>> I don't have any
Re: Kafka-Streams-Scala for Scala 3
Josep, thanks for helping with this. I was also thinking if we might need a KIP for this change. Since you had the same though, I would say, yes, let's do a KIP. @Matthias: can you prepare a KIP? You can read up on the details on the wiki page: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals If you have any questions about the process, please let us know. Thanks for pushing this forward! -Matthias On 2/8/24 8:08 AM, Matthias Berndt wrote: Hey Josep et al, I've created a ticket regarding this. https://issues.apache.org/jira/browse/KAFKA-16237 All the best, Matthias Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat : Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when your accounts are created and we'll properly set them up so you can create and assign tickets to you. Best, On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt wrote: Thanks Josep, I've applied for a JIRA account and addressed your review comments. Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat : Hi Matthias, I think for this particular case it would be worth creating a JIRA ticket for this as it's a new "feature". Regarding the change itself, I think we need to clarify how the release process would work. Right now, the script `gradlewAll` is used (which basically runs the build with Scala version 2.12 and 2.13). If I understand your changes correctly, we would need to run the build 3 times: - 1 with property scalaVersion 2.12 - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 I think we should document this and discuss when to have this feature. If I remember correctly from when I tried to update Kafka to Scala 3, the idea was to push this to a Kafka 4.0 version because we didn't want to maintain more than 2 Scala versions at the same time. I would encourage if not having a KIP, at least open up a [DISCUSS] thread to clarify some of these points. I'll add some feedback on the PR itself regarding the changes. Best, On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < matthias.ber...@ttmzero.com> wrote: Hi Matthias J., Hi Lucas, Hi Josep, Thank you for your encouraging responses regarding a Scala 3 port of Kafka-Streams-Scala, and apologies for the late response from my side. I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while retaining support for 2.13 and 2.12). Almost no changes to the code were required and the tests also pass. Please take a look and let me know what you think :-) https://github.com/apache/kafka/pull/15338 All the best Matthias Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat : Hi, For reference, prior work on this: https://github.com/apache/kafka/pull/11350 https://github.com/apache/kafka/pull/11432 Best, On Thu, Feb 1, 2024, 15:55 Lucas Brutschy .invalid> wrote: Hi Matthiases, I know Scala 2 fairly well, so I'd be happy to review changes that add Scala 3 support. However, as Matthias S. said, it has to be driven by people who use Scala day-to-day, since I believe most Kafka Streams committers are working with Java. Rewriting the tests to not use EmbeddedKafkaCluster seems like a large undertaking, so option 1 is the first thing we should explore. I don't have any experience with Scala 3 migration topics, but on the Scala website it says The first piece of good news is that the Scala 3 compiler is able to read the Scala 2.13 Pickle format and thus it can type check code that depends on modules or libraries compiled with Scala 2.13. One notable example is the Scala 2.13 library. We have indeed decided that the Scala 2.13 library is the official standard library for Scala 3. So wouldn't that mean that we are safe in terms of standard library upgrades if we use core_2.13 in the tests? Cheers, Lucas On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax wrote: Thanks for raising this. The `kafka-streams-scala` module seems to be an important feature for Kafka Streams and I am generally in favor of your proposal to add Scala 3 support. However, I am personally no Scala person and it sounds like quite some overhead. If you are willing to drive and own this initiative happy to support you to the extend I can. About the concrete proposal: my understanding is that :core will move off Scala long-term (not 100% sure what the timeline is, but new modules are written in Java only). Thus, down the road the compatibility issue would go away naturally, but it's unclear when. Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we could add support for Scala 3 now, taking a risk that it might break in the future assume that the migration off Scala from core is not fast enough. For proposal (2), I don't think that it would be easily possible for unit/integration tests. We could fall back to system tests though, but they would be much more heavy weight of course. Might be good to
Re: Kafka-Streams-Scala for Scala 3
Hey Josep et al, I've created a ticket regarding this. https://issues.apache.org/jira/browse/KAFKA-16237 All the best, Matthias Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat : > > Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when > your accounts are created and we'll properly set them up so you can create > and assign tickets to you. > > Best, > > On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt > wrote: > > > Thanks Josep, I've applied for a JIRA account and addressed your > > review comments. > > > > Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > : > > > > > > Hi Matthias, > > > > > > I think for this particular case it would be worth creating a JIRA ticket > > > for this as it's a new "feature". > > > Regarding the change itself, I think we need to clarify how the release > > > process would work. Right now, the script `gradlewAll` is used (which > > > basically runs the build with Scala version 2.12 and 2.13). If I > > understand > > > your changes correctly, we would need to run the build 3 times: > > > - 1 with property scalaVersion 2.12 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > > > I think we should document this and discuss when to have this feature. > > If I > > > remember correctly from when I tried to update Kafka to Scala 3, the idea > > > was to push this to a Kafka 4.0 version because we didn't want to > > maintain > > > more than 2 Scala versions at the same time. I would encourage if not > > > having a KIP, at least open up a [DISCUSS] thread to clarify some of > > these > > > points. > > > > > > I'll add some feedback on the PR itself regarding the changes. > > > > > > Best, > > > > > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > > matthias.ber...@ttmzero.com> > > > wrote: > > > > > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > > > > > Thank you for your encouraging responses regarding a Scala 3 port of > > > > Kafka-Streams-Scala, and apologies for the late response from my side. > > > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > > > > retaining support for 2.13 and 2.12). Almost no changes to the code > > > > were required and the tests also pass. Please take a look and let me > > > > know what you think :-) > > > > https://github.com/apache/kafka/pull/15338 > > > > > > > > All the best > > > > Matthias > > > > > > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > > > : > > > > > > > > > > Hi, > > > > > > > > > > For reference, prior work on this: > > > > > https://github.com/apache/kafka/pull/11350 > > > > > https://github.com/apache/kafka/pull/11432 > > > > > > > > > > Best, > > > > > > > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy > > > .invalid> > > > > > wrote: > > > > > > > > > > > Hi Matthiases, > > > > > > > > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that > > add > > > > > > Scala 3 support. However, as Matthias S. said, it has to be driven > > by > > > > > > people who use Scala day-to-day, since I believe most Kafka Streams > > > > > > committers are working with Java. > > > > > > > > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a > > large > > > > > > undertaking, so option 1 is the first thing we should explore. > > > > > > > > > > > > I don't have any experience with Scala 3 migration topics, but on > > the > > > > > > Scala website it says > > > > > > > The first piece of good news is that the Scala 3 compiler is > > able to > > > > > > read the Scala 2.13 Pickle format and thus it can type check code > > that > > > > > > depends on modules or libraries compiled with Scala 2.13. > > > > > > > One notable example is the Scala 2.13 library. We have indeed > > decided > > > > > > that the Scala 2.13 library is the official standard library for > > Scala > > > > 3. > > > > > > So wouldn't that mean that we are safe in terms of standard library > > > > > > upgrades if we use core_2.13 in the tests? > > > > > > > > > > > > Cheers, > > > > > > Lucas > > > > > > > > > > > > > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax > > > > wrote: > > > > > > > > > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems > > to > > > > be an > > > > > > > important feature for Kafka Streams and I am generally in favor > > of > > > > your > > > > > > > proposal to add Scala 3 support. However, I am personally no > > Scala > > > > > > > person and it sounds like quite some overhead. > > > > > > > > > > > > > > If you are willing to drive and own this initiative happy to > > support > > > > you > > > > > > > to the extend I can. > > > > > > > > > > > > > > About the concrete proposal: my understanding is that :core will > > move > > > > > > > off Scala long-term (not 100% sure what the timeline is, but new > > > > modules > > > > > > > are written in Java only). Thus, down the road the compatibility > > > > issue > > > > > > >
Re: Kafka-Streams-Scala for Scala 3
Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when your accounts are created and we'll properly set them up so you can create and assign tickets to you. Best, On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt wrote: > Thanks Josep, I've applied for a JIRA account and addressed your > review comments. > > Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > : > > > > Hi Matthias, > > > > I think for this particular case it would be worth creating a JIRA ticket > > for this as it's a new "feature". > > Regarding the change itself, I think we need to clarify how the release > > process would work. Right now, the script `gradlewAll` is used (which > > basically runs the build with Scala version 2.12 and 2.13). If I > understand > > your changes correctly, we would need to run the build 3 times: > > - 1 with property scalaVersion 2.12 > > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > > I think we should document this and discuss when to have this feature. > If I > > remember correctly from when I tried to update Kafka to Scala 3, the idea > > was to push this to a Kafka 4.0 version because we didn't want to > maintain > > more than 2 Scala versions at the same time. I would encourage if not > > having a KIP, at least open up a [DISCUSS] thread to clarify some of > these > > points. > > > > I'll add some feedback on the PR itself regarding the changes. > > > > Best, > > > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > matthias.ber...@ttmzero.com> > > wrote: > > > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > > > Thank you for your encouraging responses regarding a Scala 3 port of > > > Kafka-Streams-Scala, and apologies for the late response from my side. > > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > > > retaining support for 2.13 and 2.12). Almost no changes to the code > > > were required and the tests also pass. Please take a look and let me > > > know what you think :-) > > > https://github.com/apache/kafka/pull/15338 > > > > > > All the best > > > Matthias > > > > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > > : > > > > > > > > Hi, > > > > > > > > For reference, prior work on this: > > > > https://github.com/apache/kafka/pull/11350 > > > > https://github.com/apache/kafka/pull/11432 > > > > > > > > Best, > > > > > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy > > .invalid> > > > > wrote: > > > > > > > > > Hi Matthiases, > > > > > > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that > add > > > > > Scala 3 support. However, as Matthias S. said, it has to be driven > by > > > > > people who use Scala day-to-day, since I believe most Kafka Streams > > > > > committers are working with Java. > > > > > > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a > large > > > > > undertaking, so option 1 is the first thing we should explore. > > > > > > > > > > I don't have any experience with Scala 3 migration topics, but on > the > > > > > Scala website it says > > > > > > The first piece of good news is that the Scala 3 compiler is > able to > > > > > read the Scala 2.13 Pickle format and thus it can type check code > that > > > > > depends on modules or libraries compiled with Scala 2.13. > > > > > > One notable example is the Scala 2.13 library. We have indeed > decided > > > > > that the Scala 2.13 library is the official standard library for > Scala > > > 3. > > > > > So wouldn't that mean that we are safe in terms of standard library > > > > > upgrades if we use core_2.13 in the tests? > > > > > > > > > > Cheers, > > > > > Lucas > > > > > > > > > > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax > > > wrote: > > > > > > > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems > to > > > be an > > > > > > important feature for Kafka Streams and I am generally in favor > of > > > your > > > > > > proposal to add Scala 3 support. However, I am personally no > Scala > > > > > > person and it sounds like quite some overhead. > > > > > > > > > > > > If you are willing to drive and own this initiative happy to > support > > > you > > > > > > to the extend I can. > > > > > > > > > > > > About the concrete proposal: my understanding is that :core will > move > > > > > > off Scala long-term (not 100% sure what the timeline is, but new > > > modules > > > > > > are written in Java only). Thus, down the road the compatibility > > > issue > > > > > > would go away naturally, but it's unclear when. > > > > > > > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it > seems we > > > > > > could add support for Scala 3 now, taking a risk that it might > break > > > in > > > > > > the future assume that the migration off Scala from core is not > fast > > > > > enough. > > > > > > > > > > > > For proposal (2), I don't think that it would be easily possible > for > > > > > >
Re: Kafka-Streams-Scala for Scala 3
Thanks Josep, I've applied for a JIRA account and addressed your review comments. Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat : > > Hi Matthias, > > I think for this particular case it would be worth creating a JIRA ticket > for this as it's a new "feature". > Regarding the change itself, I think we need to clarify how the release > process would work. Right now, the script `gradlewAll` is used (which > basically runs the build with Scala version 2.12 and 2.13). If I understand > your changes correctly, we would need to run the build 3 times: > - 1 with property scalaVersion 2.12 > - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > I think we should document this and discuss when to have this feature. If I > remember correctly from when I tried to update Kafka to Scala 3, the idea > was to push this to a Kafka 4.0 version because we didn't want to maintain > more than 2 Scala versions at the same time. I would encourage if not > having a KIP, at least open up a [DISCUSS] thread to clarify some of these > points. > > I'll add some feedback on the PR itself regarding the changes. > > Best, > > On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt > wrote: > > > Hi Matthias J., Hi Lucas, Hi Josep, > > > > Thank you for your encouraging responses regarding a Scala 3 port of > > Kafka-Streams-Scala, and apologies for the late response from my side. > > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > > retaining support for 2.13 and 2.12). Almost no changes to the code > > were required and the tests also pass. Please take a look and let me > > know what you think :-) > > https://github.com/apache/kafka/pull/15338 > > > > All the best > > Matthias > > > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > : > > > > > > Hi, > > > > > > For reference, prior work on this: > > > https://github.com/apache/kafka/pull/11350 > > > https://github.com/apache/kafka/pull/11432 > > > > > > Best, > > > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy > .invalid> > > > wrote: > > > > > > > Hi Matthiases, > > > > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that add > > > > Scala 3 support. However, as Matthias S. said, it has to be driven by > > > > people who use Scala day-to-day, since I believe most Kafka Streams > > > > committers are working with Java. > > > > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large > > > > undertaking, so option 1 is the first thing we should explore. > > > > > > > > I don't have any experience with Scala 3 migration topics, but on the > > > > Scala website it says > > > > > The first piece of good news is that the Scala 3 compiler is able to > > > > read the Scala 2.13 Pickle format and thus it can type check code that > > > > depends on modules or libraries compiled with Scala 2.13. > > > > > One notable example is the Scala 2.13 library. We have indeed decided > > > > that the Scala 2.13 library is the official standard library for Scala > > 3. > > > > So wouldn't that mean that we are safe in terms of standard library > > > > upgrades if we use core_2.13 in the tests? > > > > > > > > Cheers, > > > > Lucas > > > > > > > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax > > wrote: > > > > > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems to > > be an > > > > > important feature for Kafka Streams and I am generally in favor of > > your > > > > > proposal to add Scala 3 support. However, I am personally no Scala > > > > > person and it sounds like quite some overhead. > > > > > > > > > > If you are willing to drive and own this initiative happy to support > > you > > > > > to the extend I can. > > > > > > > > > > About the concrete proposal: my understanding is that :core will move > > > > > off Scala long-term (not 100% sure what the timeline is, but new > > modules > > > > > are written in Java only). Thus, down the road the compatibility > > issue > > > > > would go away naturally, but it's unclear when. > > > > > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > > > > > could add support for Scala 3 now, taking a risk that it might break > > in > > > > > the future assume that the migration off Scala from core is not fast > > > > enough. > > > > > > > > > > For proposal (2), I don't think that it would be easily possible for > > > > > unit/integration tests. We could fall back to system tests though, > > but > > > > > they would be much more heavy weight of course. > > > > > > > > > > Might be good to hear from others. We might actually also want to do > > a > > > > > KIP for this? > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > > > > Hey there, > > > > > > > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala > > library. > > > > > > Currently, the build system is set up such that
Re: Kafka-Streams-Scala for Scala 3
Hi Matthias, I think for this particular case it would be worth creating a JIRA ticket for this as it's a new "feature". Regarding the change itself, I think we need to clarify how the release process would work. Right now, the script `gradlewAll` is used (which basically runs the build with Scala version 2.12 and 2.13). If I understand your changes correctly, we would need to run the build 3 times: - 1 with property scalaVersion 2.12 - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 I think we should document this and discuss when to have this feature. If I remember correctly from when I tried to update Kafka to Scala 3, the idea was to push this to a Kafka 4.0 version because we didn't want to maintain more than 2 Scala versions at the same time. I would encourage if not having a KIP, at least open up a [DISCUSS] thread to clarify some of these points. I'll add some feedback on the PR itself regarding the changes. Best, On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt wrote: > Hi Matthias J., Hi Lucas, Hi Josep, > > Thank you for your encouraging responses regarding a Scala 3 port of > Kafka-Streams-Scala, and apologies for the late response from my side. > I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while > retaining support for 2.13 and 2.12). Almost no changes to the code > were required and the tests also pass. Please take a look and let me > know what you think :-) > https://github.com/apache/kafka/pull/15338 > > All the best > Matthias > > Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > : > > > > Hi, > > > > For reference, prior work on this: > > https://github.com/apache/kafka/pull/11350 > > https://github.com/apache/kafka/pull/11432 > > > > Best, > > > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy .invalid> > > wrote: > > > > > Hi Matthiases, > > > > > > I know Scala 2 fairly well, so I'd be happy to review changes that add > > > Scala 3 support. However, as Matthias S. said, it has to be driven by > > > people who use Scala day-to-day, since I believe most Kafka Streams > > > committers are working with Java. > > > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large > > > undertaking, so option 1 is the first thing we should explore. > > > > > > I don't have any experience with Scala 3 migration topics, but on the > > > Scala website it says > > > > The first piece of good news is that the Scala 3 compiler is able to > > > read the Scala 2.13 Pickle format and thus it can type check code that > > > depends on modules or libraries compiled with Scala 2.13. > > > > One notable example is the Scala 2.13 library. We have indeed decided > > > that the Scala 2.13 library is the official standard library for Scala > 3. > > > So wouldn't that mean that we are safe in terms of standard library > > > upgrades if we use core_2.13 in the tests? > > > > > > Cheers, > > > Lucas > > > > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax > wrote: > > > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems to > be an > > > > important feature for Kafka Streams and I am generally in favor of > your > > > > proposal to add Scala 3 support. However, I am personally no Scala > > > > person and it sounds like quite some overhead. > > > > > > > > If you are willing to drive and own this initiative happy to support > you > > > > to the extend I can. > > > > > > > > About the concrete proposal: my understanding is that :core will move > > > > off Scala long-term (not 100% sure what the timeline is, but new > modules > > > > are written in Java only). Thus, down the road the compatibility > issue > > > > would go away naturally, but it's unclear when. > > > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > > > > could add support for Scala 3 now, taking a risk that it might break > in > > > > the future assume that the migration off Scala from core is not fast > > > enough. > > > > > > > > For proposal (2), I don't think that it would be easily possible for > > > > unit/integration tests. We could fall back to system tests though, > but > > > > they would be much more heavy weight of course. > > > > > > > > Might be good to hear from others. We might actually also want to do > a > > > > KIP for this? > > > > > > > > > > > > -Matthias > > > > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > > > Hey there, > > > > > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala > library. > > > > > Currently, the build system is set up such that kafka-streams-scala > > > > > and core (i. e. kafka itself) are compiled with the same Scala > > > > > compiler versions. This is not an optimal situation because it > means > > > > > that a Scala 3 release of kafka-streams-scala cannot happen > > > > > independently of kafka itself. I think this should be changed > > > > > > > > > > The production codebase of scala-streams-kafka actually
Re: Kafka-Streams-Scala for Scala 3
Hi Matthias J., Hi Lucas, Hi Josep, Thank you for your encouraging responses regarding a Scala 3 port of Kafka-Streams-Scala, and apologies for the late response from my side. I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while retaining support for 2.13 and 2.12). Almost no changes to the code were required and the tests also pass. Please take a look and let me know what you think :-) https://github.com/apache/kafka/pull/15338 All the best Matthias Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat : > > Hi, > > For reference, prior work on this: > https://github.com/apache/kafka/pull/11350 > https://github.com/apache/kafka/pull/11432 > > Best, > > On Thu, Feb 1, 2024, 15:55 Lucas Brutschy > wrote: > > > Hi Matthiases, > > > > I know Scala 2 fairly well, so I'd be happy to review changes that add > > Scala 3 support. However, as Matthias S. said, it has to be driven by > > people who use Scala day-to-day, since I believe most Kafka Streams > > committers are working with Java. > > > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large > > undertaking, so option 1 is the first thing we should explore. > > > > I don't have any experience with Scala 3 migration topics, but on the > > Scala website it says > > > The first piece of good news is that the Scala 3 compiler is able to > > read the Scala 2.13 Pickle format and thus it can type check code that > > depends on modules or libraries compiled with Scala 2.13. > > > One notable example is the Scala 2.13 library. We have indeed decided > > that the Scala 2.13 library is the official standard library for Scala 3. > > So wouldn't that mean that we are safe in terms of standard library > > upgrades if we use core_2.13 in the tests? > > > > Cheers, > > Lucas > > > > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax wrote: > > > > > > Thanks for raising this. The `kafka-streams-scala` module seems to be an > > > important feature for Kafka Streams and I am generally in favor of your > > > proposal to add Scala 3 support. However, I am personally no Scala > > > person and it sounds like quite some overhead. > > > > > > If you are willing to drive and own this initiative happy to support you > > > to the extend I can. > > > > > > About the concrete proposal: my understanding is that :core will move > > > off Scala long-term (not 100% sure what the timeline is, but new modules > > > are written in Java only). Thus, down the road the compatibility issue > > > would go away naturally, but it's unclear when. > > > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > > > could add support for Scala 3 now, taking a risk that it might break in > > > the future assume that the migration off Scala from core is not fast > > enough. > > > > > > For proposal (2), I don't think that it would be easily possible for > > > unit/integration tests. We could fall back to system tests though, but > > > they would be much more heavy weight of course. > > > > > > Might be good to hear from others. We might actually also want to do a > > > KIP for this? > > > > > > > > > -Matthias > > > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > > Hey there, > > > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library. > > > > Currently, the build system is set up such that kafka-streams-scala > > > > and core (i. e. kafka itself) are compiled with the same Scala > > > > compiler versions. This is not an optimal situation because it means > > > > that a Scala 3 release of kafka-streams-scala cannot happen > > > > independently of kafka itself. I think this should be changed > > > > > > > > The production codebase of scala-streams-kafka actually compiles just > > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The > > > > problem is with the tests. These use the `EmbeddedKafkaCluster` class, > > > > which means that kafka is pulled into the classpath, potentially > > > > leading to binary compatibility issues. > > > > I can see several approaches to fixing this: > > > > > > > > 1. Run the kafka-streams-scala tests using the compatible version of > > > > :core if one is available. Currently, this means that everything can > > > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 > > > > using core_2.13, as these should be compatible), but when a new > > > > scala-library version is released that is no longer compatible with > > > > 2.13, we won't be able to test that. > > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead > > > > running the test cluster in a separate JVM or perhaps even a > > > > container. > > > > > > > > I'd be willing to get my hands dirty working on this, but before I > > > > start I'd like to get some feedback from the Kafka team regarding the > > > > approaches outlined above. > > > > > > > > All the best > > > > Matthias Berndt > > > KJosep Prat > Open
Re: Kafka-Streams-Scala for Scala 3
Hi, For reference, prior work on this: https://github.com/apache/kafka/pull/11350 https://github.com/apache/kafka/pull/11432 Best, On Thu, Feb 1, 2024, 15:55 Lucas Brutschy wrote: > Hi Matthiases, > > I know Scala 2 fairly well, so I'd be happy to review changes that add > Scala 3 support. However, as Matthias S. said, it has to be driven by > people who use Scala day-to-day, since I believe most Kafka Streams > committers are working with Java. > > Rewriting the tests to not use EmbeddedKafkaCluster seems like a large > undertaking, so option 1 is the first thing we should explore. > > I don't have any experience with Scala 3 migration topics, but on the > Scala website it says > > The first piece of good news is that the Scala 3 compiler is able to > read the Scala 2.13 Pickle format and thus it can type check code that > depends on modules or libraries compiled with Scala 2.13. > > One notable example is the Scala 2.13 library. We have indeed decided > that the Scala 2.13 library is the official standard library for Scala 3. > So wouldn't that mean that we are safe in terms of standard library > upgrades if we use core_2.13 in the tests? > > Cheers, > Lucas > > > On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax wrote: > > > > Thanks for raising this. The `kafka-streams-scala` module seems to be an > > important feature for Kafka Streams and I am generally in favor of your > > proposal to add Scala 3 support. However, I am personally no Scala > > person and it sounds like quite some overhead. > > > > If you are willing to drive and own this initiative happy to support you > > to the extend I can. > > > > About the concrete proposal: my understanding is that :core will move > > off Scala long-term (not 100% sure what the timeline is, but new modules > > are written in Java only). Thus, down the road the compatibility issue > > would go away naturally, but it's unclear when. > > > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > > could add support for Scala 3 now, taking a risk that it might break in > > the future assume that the migration off Scala from core is not fast > enough. > > > > For proposal (2), I don't think that it would be easily possible for > > unit/integration tests. We could fall back to system tests though, but > > they would be much more heavy weight of course. > > > > Might be good to hear from others. We might actually also want to do a > > KIP for this? > > > > > > -Matthias > > > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > Hey there, > > > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library. > > > Currently, the build system is set up such that kafka-streams-scala > > > and core (i. e. kafka itself) are compiled with the same Scala > > > compiler versions. This is not an optimal situation because it means > > > that a Scala 3 release of kafka-streams-scala cannot happen > > > independently of kafka itself. I think this should be changed > > > > > > The production codebase of scala-streams-kafka actually compiles just > > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The > > > problem is with the tests. These use the `EmbeddedKafkaCluster` class, > > > which means that kafka is pulled into the classpath, potentially > > > leading to binary compatibility issues. > > > I can see several approaches to fixing this: > > > > > > 1. Run the kafka-streams-scala tests using the compatible version of > > > :core if one is available. Currently, this means that everything can > > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 > > > using core_2.13, as these should be compatible), but when a new > > > scala-library version is released that is no longer compatible with > > > 2.13, we won't be able to test that. > > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead > > > running the test cluster in a separate JVM or perhaps even a > > > container. > > > > > > I'd be willing to get my hands dirty working on this, but before I > > > start I'd like to get some feedback from the Kafka team regarding the > > > approaches outlined above. > > > > > > All the best > > > Matthias Berndt > KJosep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: Kafka-Streams-Scala for Scala 3
Hi Matthiases, I know Scala 2 fairly well, so I'd be happy to review changes that add Scala 3 support. However, as Matthias S. said, it has to be driven by people who use Scala day-to-day, since I believe most Kafka Streams committers are working with Java. Rewriting the tests to not use EmbeddedKafkaCluster seems like a large undertaking, so option 1 is the first thing we should explore. I don't have any experience with Scala 3 migration topics, but on the Scala website it says > The first piece of good news is that the Scala 3 compiler is able to read the > Scala 2.13 Pickle format and thus it can type check code that depends on > modules or libraries compiled with Scala 2.13. > One notable example is the Scala 2.13 library. We have indeed decided that > the Scala 2.13 library is the official standard library for Scala 3. So wouldn't that mean that we are safe in terms of standard library upgrades if we use core_2.13 in the tests? Cheers, Lucas On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax wrote: > > Thanks for raising this. The `kafka-streams-scala` module seems to be an > important feature for Kafka Streams and I am generally in favor of your > proposal to add Scala 3 support. However, I am personally no Scala > person and it sounds like quite some overhead. > > If you are willing to drive and own this initiative happy to support you > to the extend I can. > > About the concrete proposal: my understanding is that :core will move > off Scala long-term (not 100% sure what the timeline is, but new modules > are written in Java only). Thus, down the road the compatibility issue > would go away naturally, but it's unclear when. > > Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we > could add support for Scala 3 now, taking a risk that it might break in > the future assume that the migration off Scala from core is not fast enough. > > For proposal (2), I don't think that it would be easily possible for > unit/integration tests. We could fall back to system tests though, but > they would be much more heavy weight of course. > > Might be good to hear from others. We might actually also want to do a > KIP for this? > > > -Matthias > > On 1/20/24 10:34 AM, Matthias Berndt wrote: > > Hey there, > > > > I'd like to discuss a Scala 3 port of the kafka-streams-scala library. > > Currently, the build system is set up such that kafka-streams-scala > > and core (i. e. kafka itself) are compiled with the same Scala > > compiler versions. This is not an optimal situation because it means > > that a Scala 3 release of kafka-streams-scala cannot happen > > independently of kafka itself. I think this should be changed > > > > The production codebase of scala-streams-kafka actually compiles just > > fine on Scala 3.3.1 with two lines of trivial syntax changes. The > > problem is with the tests. These use the `EmbeddedKafkaCluster` class, > > which means that kafka is pulled into the classpath, potentially > > leading to binary compatibility issues. > > I can see several approaches to fixing this: > > > > 1. Run the kafka-streams-scala tests using the compatible version of > > :core if one is available. Currently, this means that everything can > > be tested (test kafka-streams-scala_2.12 using core_2.12, > > kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 > > using core_2.13, as these should be compatible), but when a new > > scala-library version is released that is no longer compatible with > > 2.13, we won't be able to test that. > > 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead > > running the test cluster in a separate JVM or perhaps even a > > container. > > > > I'd be willing to get my hands dirty working on this, but before I > > start I'd like to get some feedback from the Kafka team regarding the > > approaches outlined above. > > > > All the best > > Matthias Berndt
Re: Kafka-Streams-Scala for Scala 3
Thanks for raising this. The `kafka-streams-scala` module seems to be an important feature for Kafka Streams and I am generally in favor of your proposal to add Scala 3 support. However, I am personally no Scala person and it sounds like quite some overhead. If you are willing to drive and own this initiative happy to support you to the extend I can. About the concrete proposal: my understanding is that :core will move off Scala long-term (not 100% sure what the timeline is, but new modules are written in Java only). Thus, down the road the compatibility issue would go away naturally, but it's unclear when. Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we could add support for Scala 3 now, taking a risk that it might break in the future assume that the migration off Scala from core is not fast enough. For proposal (2), I don't think that it would be easily possible for unit/integration tests. We could fall back to system tests though, but they would be much more heavy weight of course. Might be good to hear from others. We might actually also want to do a KIP for this? -Matthias On 1/20/24 10:34 AM, Matthias Berndt wrote: Hey there, I'd like to discuss a Scala 3 port of the kafka-streams-scala library. Currently, the build system is set up such that kafka-streams-scala and core (i. e. kafka itself) are compiled with the same Scala compiler versions. This is not an optimal situation because it means that a Scala 3 release of kafka-streams-scala cannot happen independently of kafka itself. I think this should be changed The production codebase of scala-streams-kafka actually compiles just fine on Scala 3.3.1 with two lines of trivial syntax changes. The problem is with the tests. These use the `EmbeddedKafkaCluster` class, which means that kafka is pulled into the classpath, potentially leading to binary compatibility issues. I can see several approaches to fixing this: 1. Run the kafka-streams-scala tests using the compatible version of :core if one is available. Currently, this means that everything can be tested (test kafka-streams-scala_2.12 using core_2.12, kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3 using core_2.13, as these should be compatible), but when a new scala-library version is released that is no longer compatible with 2.13, we won't be able to test that. 2. Rewrite the tests to run without EmbeddedKafkaCluster, instead running the test cluster in a separate JVM or perhaps even a container. I'd be willing to get my hands dirty working on this, but before I start I'd like to get some feedback from the Kafka team regarding the approaches outlined above. All the best Matthias Berndt
