Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Hi Ismael, Thanks for the feedback, I can definitely raise a KIP, that is no problem. I will write one up and then we can have further discussion on the details. I should have time to get one created later today or by tomorrow. Chris On Wed, Mar 27, 2024 at 11:54 AM Ismael Juma wrote: > Hi Christopher, > > Thanks for raising this. Moving to the new namespace makes sense - would > you be willing to submit a KIP? The point you raised regarding Jetty 11 EOL > and Jetty 12 requiring Java 17 is a good one and is worth discussing the > trade-offs in more detail. I originally did not propose moving Connect to > Java 17 because of the risk that it might break several connectors. If > someone summarized the number of connectors that support Java 17 and the > number that does not, it would be a useful first step in the discussion. > > Ismael > > On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon < > christopher.l.shan...@gmail.com> wrote: > > > Hi Greg, > > > > You are right that KIP-1013 and the JDK 17 upgrade is not directly > relevant > > because JDK 11 can still be used with Jakarta APIs. However, it's still > > somewhat relevant and important because if we are stuck at JDK 11 then we > > can't upgrade to certain versions. For Connect, there is a Jetty server > for > > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The > > problem with using Jetty 11.x is that it is already EOL. > > > > So are we really locked into JDK 11 for Connect for version 4.0? It would > > require people to upgrade their connectors to run on JDK 17 but shipping > > Kafka 4.0 with a Jetty version that is already end of life doesn't make > > sense to me. I know that Connect supports isolated classloaders for > > connectors but that of course is not the same as different Java versions. > > > > Chris > > > > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris > > > > > wrote: > > > > > Hi Christopher! > > > > > > Thanks so much for raising this. I agree that we should move to the > > > new namespace in 4.0, and not doing so would be a mistake. > > > This breaking change has a lot of benefits, and the only cost I am > > > aware of is that ConnectRestExtensions will need to be migrated, > > > rebuilt, and re-released for 4.0+ > > > > > > Can you explain how KIP-1013 and the Java version are relevant? > > > Connect is dependent on this namespace, but will still need to support > > > Java 11 in 4.0. > > > > > > Thanks! > > > Greg > > > > > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon > > > wrote: > > > > > > > > Is this already being planned for version 4.0? If not, I strongly > thing > > > it > > > > should be. > > > > > > > > Kafka is currently using the old long deprecated javax apis which is > > > going > > > > to continue to cause issues [1] for people as more and more things > are > > > > updated to use Jakarta. > > > > > > > > With the bump to require JDK 17 for version 4.0 [2] this seems like > the > > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis > and > > > new > > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that > > all > > > > support the new namespace. It needs to be upgraded at some point > > anyways > > > so > > > > a major version makes sense to me. > > > > > > > > Another scenario where I've run into this problem is testing. For > > > example, > > > > If I try to run tests against my custom code with an embedded Kafka > > > broker > > > > and components in JUnit, then things break with newer dependencies > like > > > > Spring that require Jakarta as it interferes on the classpath. > > > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > > > > [2] > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 > > > > > >
Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Hi Christopher, Thanks for raising this. Moving to the new namespace makes sense - would you be willing to submit a KIP? The point you raised regarding Jetty 11 EOL and Jetty 12 requiring Java 17 is a good one and is worth discussing the trade-offs in more detail. I originally did not propose moving Connect to Java 17 because of the risk that it might break several connectors. If someone summarized the number of connectors that support Java 17 and the number that does not, it would be a useful first step in the discussion. Ismael On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon < christopher.l.shan...@gmail.com> wrote: > Hi Greg, > > You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant > because JDK 11 can still be used with Jakarta APIs. However, it's still > somewhat relevant and important because if we are stuck at JDK 11 then we > can't upgrade to certain versions. For Connect, there is a Jetty server for > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The > problem with using Jetty 11.x is that it is already EOL. > > So are we really locked into JDK 11 for Connect for version 4.0? It would > require people to upgrade their connectors to run on JDK 17 but shipping > Kafka 4.0 with a Jetty version that is already end of life doesn't make > sense to me. I know that Connect supports isolated classloaders for > connectors but that of course is not the same as different Java versions. > > Chris > > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris > > wrote: > > > Hi Christopher! > > > > Thanks so much for raising this. I agree that we should move to the > > new namespace in 4.0, and not doing so would be a mistake. > > This breaking change has a lot of benefits, and the only cost I am > > aware of is that ConnectRestExtensions will need to be migrated, > > rebuilt, and re-released for 4.0+ > > > > Can you explain how KIP-1013 and the Java version are relevant? > > Connect is dependent on this namespace, but will still need to support > > Java 11 in 4.0. > > > > Thanks! > > Greg > > > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon > > wrote: > > > > > > Is this already being planned for version 4.0? If not, I strongly thing > > it > > > should be. > > > > > > Kafka is currently using the old long deprecated javax apis which is > > going > > > to continue to cause issues [1] for people as more and more things are > > > updated to use Jakarta. > > > > > > With the bump to require JDK 17 for version 4.0 [2] this seems like the > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis and > > new > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that > all > > > support the new namespace. It needs to be upgraded at some point > anyways > > so > > > a major version makes sense to me. > > > > > > Another scenario where I've run into this problem is testing. For > > example, > > > If I try to run tests against my custom code with an embedded Kafka > > broker > > > and components in JUnit, then things break with newer dependencies like > > > Spring that require Jakarta as it interferes on the classpath. > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > > > [2] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 > > >
Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Greg, I definitely understand that part, it's always tricky for users that have older software and haven't or can't upgrade yet. The most important thing to me is getting to the new Jakarta APIs and Java EE updates as that causes incompatibilities with code and requires code changes. So if that requires going with Jetty 11.x for now, that is better than sticking with 9.4.x at least. If someone wanted to run Jetty 12 I wouldn't expect it to be a huge deal for someone if they really wanted to do that and swap it out (especially if using Maven or Gradle to create a custom build) as it wouldn't require code changes. I just checked again and it looks like while Jetty did end community support for 11.x in January, they are at least still providing security fixes for 2024 [1] and maybe beyond which is the most important thing. It would be a terrible idea to include something like a web server that wouldn't receive CVE updates. I am not sure without diving in what else would need to be upgraded for but it looks like there's actually an old PR [2] that was closed to migrate to Jetty 11 and Jakarta (as well as other necessary updates like Jersey) maybe that could be reopened? Also, I assume this would need a KIP to proceed. Chris [1] https://github.com/jetty/jetty.project/issues/10485 [2] https://github.com/apache/kafka/pull/10176 On Tue, Mar 26, 2024 at 1:11 PM Greg Harris wrote: > Hey Christopher, > > Thank you for explaining that jetty version dependency, I was not aware of > that. > > > So are we really locked into JDK 11 for Connect for version 4.0? > > Until we adopt a KIP to change the minimum version, we must support 11 > in 4.0. By default we will continue to support Java versions after a > major version upgrade. > > There could be some friction for trying to get such a KIP passed given > the ecosystem of connectors, their maintenance practices, and how soon > 4.0 is planned. If it doesn't get adopted, we could also use this > version dependency as a motivation for having 5.0 sooner rather than > later, and use the intervening time to properly notify downstream > users to upgrade their Java versions. > > Thanks, > Greg > > On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon > wrote: > > > > Hi Greg, > > > > You are right that KIP-1013 and the JDK 17 upgrade is not directly > relevant > > because JDK 11 can still be used with Jakarta APIs. However, it's still > > somewhat relevant and important because if we are stuck at JDK 11 then we > > can't upgrade to certain versions. For Connect, there is a Jetty server > for > > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The > > problem with using Jetty 11.x is that it is already EOL. > > > > So are we really locked into JDK 11 for Connect for version 4.0? It would > > require people to upgrade their connectors to run on JDK 17 but shipping > > Kafka 4.0 with a Jetty version that is already end of life doesn't make > > sense to me. I know that Connect supports isolated classloaders for > > connectors but that of course is not the same as different Java versions. > > > > Chris > > > > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris > > > wrote: > > > > > Hi Christopher! > > > > > > Thanks so much for raising this. I agree that we should move to the > > > new namespace in 4.0, and not doing so would be a mistake. > > > This breaking change has a lot of benefits, and the only cost I am > > > aware of is that ConnectRestExtensions will need to be migrated, > > > rebuilt, and re-released for 4.0+ > > > > > > Can you explain how KIP-1013 and the Java version are relevant? > > > Connect is dependent on this namespace, but will still need to support > > > Java 11 in 4.0. > > > > > > Thanks! > > > Greg > > > > > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon > > > wrote: > > > > > > > > Is this already being planned for version 4.0? If not, I strongly > thing > > > it > > > > should be. > > > > > > > > Kafka is currently using the old long deprecated javax apis which is > > > going > > > > to continue to cause issues [1] for people as more and more things > are > > > > updated to use Jakarta. > > > > > > > > With the bump to require JDK 17 for version 4.0 [2] this seems like > the > > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis > and > > > new > > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc > that all > > > > support the new namespace. It needs to be upgraded at some point > anyways > > > so > > > > a major version makes sense to me. > > > > > > > > Another scenario where I've run into this problem is testing. For > > > example, > > > > If I try to run tests against my custom code with an embedded Kafka > > > broker > > > > and components in JUnit, then things break with newer dependencies > like > > > > Spring that require Jakarta as it interferes on the classpath. > > > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > > > > [2] > > > > > > > >
Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Hey Christopher, Thank you for explaining that jetty version dependency, I was not aware of that. > So are we really locked into JDK 11 for Connect for version 4.0? Until we adopt a KIP to change the minimum version, we must support 11 in 4.0. By default we will continue to support Java versions after a major version upgrade. There could be some friction for trying to get such a KIP passed given the ecosystem of connectors, their maintenance practices, and how soon 4.0 is planned. If it doesn't get adopted, we could also use this version dependency as a motivation for having 5.0 sooner rather than later, and use the intervening time to properly notify downstream users to upgrade their Java versions. Thanks, Greg On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon wrote: > > Hi Greg, > > You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant > because JDK 11 can still be used with Jakarta APIs. However, it's still > somewhat relevant and important because if we are stuck at JDK 11 then we > can't upgrade to certain versions. For Connect, there is a Jetty server for > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The > problem with using Jetty 11.x is that it is already EOL. > > So are we really locked into JDK 11 for Connect for version 4.0? It would > require people to upgrade their connectors to run on JDK 17 but shipping > Kafka 4.0 with a Jetty version that is already end of life doesn't make > sense to me. I know that Connect supports isolated classloaders for > connectors but that of course is not the same as different Java versions. > > Chris > > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris > wrote: > > > Hi Christopher! > > > > Thanks so much for raising this. I agree that we should move to the > > new namespace in 4.0, and not doing so would be a mistake. > > This breaking change has a lot of benefits, and the only cost I am > > aware of is that ConnectRestExtensions will need to be migrated, > > rebuilt, and re-released for 4.0+ > > > > Can you explain how KIP-1013 and the Java version are relevant? > > Connect is dependent on this namespace, but will still need to support > > Java 11 in 4.0. > > > > Thanks! > > Greg > > > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon > > wrote: > > > > > > Is this already being planned for version 4.0? If not, I strongly thing > > it > > > should be. > > > > > > Kafka is currently using the old long deprecated javax apis which is > > going > > > to continue to cause issues [1] for people as more and more things are > > > updated to use Jakarta. > > > > > > With the bump to require JDK 17 for version 4.0 [2] this seems like the > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis and > > new > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all > > > support the new namespace. It needs to be upgraded at some point anyways > > so > > > a major version makes sense to me. > > > > > > Another scenario where I've run into this problem is testing. For > > example, > > > If I try to run tests against my custom code with an embedded Kafka > > broker > > > and components in JUnit, then things break with newer dependencies like > > > Spring that require Jakarta as it interferes on the classpath. > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > > > [2] > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 > >
Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Hi Greg, You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant because JDK 11 can still be used with Jakarta APIs. However, it's still somewhat relevant and important because if we are stuck at JDK 11 then we can't upgrade to certain versions. For Connect, there is a Jetty server for the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The problem with using Jetty 11.x is that it is already EOL. So are we really locked into JDK 11 for Connect for version 4.0? It would require people to upgrade their connectors to run on JDK 17 but shipping Kafka 4.0 with a Jetty version that is already end of life doesn't make sense to me. I know that Connect supports isolated classloaders for connectors but that of course is not the same as different Java versions. Chris On Tue, Mar 26, 2024 at 11:33 AM Greg Harris wrote: > Hi Christopher! > > Thanks so much for raising this. I agree that we should move to the > new namespace in 4.0, and not doing so would be a mistake. > This breaking change has a lot of benefits, and the only cost I am > aware of is that ConnectRestExtensions will need to be migrated, > rebuilt, and re-released for 4.0+ > > Can you explain how KIP-1013 and the Java version are relevant? > Connect is dependent on this namespace, but will still need to support > Java 11 in 4.0. > > Thanks! > Greg > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon > wrote: > > > > Is this already being planned for version 4.0? If not, I strongly thing > it > > should be. > > > > Kafka is currently using the old long deprecated javax apis which is > going > > to continue to cause issues [1] for people as more and more things are > > updated to use Jakarta. > > > > With the bump to require JDK 17 for version 4.0 [2] this seems like the > > perfect time to upgrade to a new version of JavaEE and Jakarta apis and > new > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all > > support the new namespace. It needs to be upgraded at some point anyways > so > > a major version makes sense to me. > > > > Another scenario where I've run into this problem is testing. For > example, > > If I try to run tests against my custom code with an embedded Kafka > broker > > and components in JUnit, then things break with newer dependencies like > > Spring that require Jakarta as it interferes on the classpath. > > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > > [2] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 >
Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Hi Christopher! Thanks so much for raising this. I agree that we should move to the new namespace in 4.0, and not doing so would be a mistake. This breaking change has a lot of benefits, and the only cost I am aware of is that ConnectRestExtensions will need to be migrated, rebuilt, and re-released for 4.0+ Can you explain how KIP-1013 and the Java version are relevant? Connect is dependent on this namespace, but will still need to support Java 11 in 4.0. Thanks! Greg On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon wrote: > > Is this already being planned for version 4.0? If not, I strongly thing it > should be. > > Kafka is currently using the old long deprecated javax apis which is going > to continue to cause issues [1] for people as more and more things are > updated to use Jakarta. > > With the bump to require JDK 17 for version 4.0 [2] this seems like the > perfect time to upgrade to a new version of JavaEE and Jakarta apis and new > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all > support the new namespace. It needs to be upgraded at some point anyways so > a major version makes sense to me. > > Another scenario where I've run into this problem is testing. For example, > If I try to run tests against my custom code with an embedded Kafka broker > and components in JUnit, then things break with newer dependencies like > Spring that require Jakarta as it interferes on the classpath. > > [1] https://issues.apache.org/jira/browse/KAFKA-16326 > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
[DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0
Is this already being planned for version 4.0? If not, I strongly thing it should be. Kafka is currently using the old long deprecated javax apis which is going to continue to cause issues [1] for people as more and more things are updated to use Jakarta. With the bump to require JDK 17 for version 4.0 [2] this seems like the perfect time to upgrade to a new version of JavaEE and Jakarta apis and new versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all support the new namespace. It needs to be upgraded at some point anyways so a major version makes sense to me. Another scenario where I've run into this problem is testing. For example, If I try to run tests against my custom code with an embedded Kafka broker and components in JUnit, then things break with newer dependencies like Spring that require Jakarta as it interferes on the classpath. [1] https://issues.apache.org/jira/browse/KAFKA-16326 [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1891670491 > @fpapon I guess you mean Kafka PMC :) Not Karaf :) Ah yes, sorry, split brain :) I fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
jbonofre commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1891649260 @fpapon I guess you mean Kafka PMC :) Not Karaf :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1891642675 As the Apache Karaf PMC rejeted this proposal on the mailing list, I'm closing this PR. Link to the thread: https://lists.apache.org/thread/sld85ly7fvvvlb5bh2856qqcpnckg2on -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon closed pull request #577: Add get support page URL: https://github.com/apache/kafka-site/pull/577 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1885101241 @ableegoldman I pushed some changes, feel free to review/comment :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884961276 @mimaison ok thanks for your feeedbacks. I will update the PR according to the comments of the people and then resend an email to the mailing for the proposal and see how the community react on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
mimaison commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884814599 No worries, there's no need to blame anybody for such a small issue. The initiative to add this page is, in my opinion, good and contributions are always welcome. It's just the initial content and quick approvals that looked concerning. @fpapon As pointed by several committers there are a few issues with the current content but we should be able to agree on what to do and update this PR accordingly. I wonder if the first step would be to decide whether we want this page on the website. It's probably best to email the dev to get some agreement, or at least allow people to raise objections and then start gathering content. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
jbonofre commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884745111 @mimaison I take the blame on me: I did mistakes with the bright side of things. 1. I thought this kind of change was approved by the community/PMC 2. As the same page exists in several ASF projects, and it was a copy from Camel, I thought it was ok straight forward without checking the diff in details. I share your points, but I would not be too sharp. @fpapon is just trying to help. I would add more companies/providers in this page as part of this PR. I think we can say that it's for the good of the Kafka community and diversity. I'm happy to chat with you about that on Slack or on a call if you want. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884696661 @mimaison I replied to the original thread on the mailing list to rebirth the discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884677270 > You have to admit this PR raised quite a few red flags! > > First it's directly copied from Camel without replacing mentions to Kafka. Then within minutes it's approved by 2 other ASF members clearly without looking at the diff. Finally it's adding your company as the sole provider for commercial support. > > A more open approach would have been to reach out the Kafka community on the dev or users lists and discuss a nice way to build such a page. I tend to agree that a page listing commercial offerings is helpful but I'm sure you understand why we can't merge the PR as is. Hi @mimaison, I admitted that I made a mistake about copy/paste from Camel website and I fixed it. About adding my company, as I explained in other comments, it's just a starting point to list companies. I cannot add the others for trademark purpose because I'm not owning them but everybody can add their company as explain in the text or I'm open to do it if people ask me in this PR. About the approach, I already start a thread on the mailing list about this topic before doing a proposal, here the link to thread: https://lists.apache.org/thread/kkox33rhtjcdr5zztq3lzj7c5s7k9wsr Sorry, it was last year, so may be I would have to resend it again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
mimaison commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884671597 As I said, I think a page listing commercial offerings is useful. I was just commenting on the approach you took. I don't think we necessarily need a committer to build this page (obviously a committer will have to review the PR before merging). If you want you can start the discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
rmannibucau commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884622256 @mimaison well I wouldn't say red flags but I fully understand the surprise on kafka side if it was never discussed on/offline first - we had the same debate on TomEE side years ago. But it is not uncommon at apache to have such a page, even with a single company or even with no company and request for companies: * https://hop.apache.org/community/commercial/ (0) * https://struts.apache.org/commercial-support.html (1) * https://directory.apache.org/commercial-support.html (1) * https://superset.apache.org/community (direct link to the company behind - this one is more surprising for me since instead of encouraging the "registration" of vendors it hides it somehow but guess it is a small clumsiness) * https://tomee.apache.org/commercial-support.html (2) * https://plc4x.apache.org/users/commercial-support.html (2) * https://camel.apache.org/community/support/ * https://openmeetings.apache.org/commercial-support.html * https://guacamole.apache.org/support/ * https://cwiki.apache.org/confluence/display/HADOOP2/Distributions+and+Commercial+Support * https://activemq.apache.org/support * https://netbeans.apache.org/front/main/help/commercial-support/ * https://royale.apache.org/royale-commercial-support/ * (way more but guess you got the idea) To give an example of the "rationale behind" please have a look to https://github.com/apache/superset/issues/8852 issue. An important point to take into consideration is that several products - and I think Kafka is exactly there - wouldn't live without external support (it is true for most broker or server) so it is guaranteeing its community and helping its user base to do this kind of page - and trust me, ~5 years ago I didn't understand that as well as today so I say it very humbly. So overall, due to kafka adoption it can only be good to get such a page IMHO but I agree that if you feel it is not straight forward and it must be discussed more deeply cause Kafka wants to write its own content, a thread sounds the way to move it forward and a core commiter should then probably take the leadership on that "doc" track and this pr marked as pending while this work is done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
mimaison commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884592204 You have to admit this PR raised quite a few red flags! First it's directly copied from Camel without replacing mentions to Kafka. Then within minutes it's approved by 2 other ASF members clearly without looking at the diff. Finally it's adding your company as the sole provider for commercial support. A more open approach would have been to reach out the Kafka community on the dev or users lists and discuss a nice way to build such a page. I tend to agree that a page listing commercial offerings is helpful but I'm sure you understand why we can't merge the PR as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884502000 > @fpapon , thanks for the PR. Could you explain the motivation that why we should add `Get Support` page? Why is the `Contact Us` page not enough? Thanks. - This page explain more how to get supported. @ableegoldman didn't said that it's weird to have commercial support list, but it's weird to have only one company listed, that I'm agree and we can ask on the mailing list if some other company want to be listed. - Agree for the vendor neutral slack channel, that's why I proposed to list the existing official ASF Kafka slack channel. About StackOverFlow, as there is a lot of content on that channel, it can be used as a FAQ for the user but users can also ask on the mailing list. I think users can search content more easily on StackOverFlow rather than on the mailing list thread history. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1884372269 @showuon There is a lot of ASF projects that provide a page with community support, it help the adoption and the growth of the project because the users need commercial support for production (SLAs), consulting or training. So if some company are listed in the project website it can be very useful for them to find support. You can check Camel, ActiveMQ, Karaf websites for example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
rmannibucau commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1447016147 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: For one rational (coming from other ASF projects): asking for help always require some sort of registration and when a company just want support (often SLA related or expertise related) this is perceived as negative so several ASF projects (most of the ones able to run as standalone, ie not libs) started to add such a page or at least point to a page enabling to find out the info. hope it helps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
showuon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446975450 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. Review Comment: I know @ableegoldman is trying to list all helpful resources here, but I don't think adding `Confluent Community Slack` or `StackOverflow` is appropriate here. `Confluent Community Slack` is run by Confluent, so it is not a neutral recourse. About the `StackOverflow`, I think if we list `StackOverflow` here, does that imply `us...@apache.kafka.org` or `d...@apache.kafka.org` are not useful? I think, as a community, we should recommend users to ask/answer questions in the community, instead of in the 3rd party site. ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: I think you should start a discussion thread to ask if we "should" have the "Commercial support" section first. I agree with @ableegoldman that it's weird the Apache Kafka site contains any commercial content. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446950508 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: @ableegoldman do you think we can start a thread on the mailing list to ask which company want to be add in the commercial list support? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446949226 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: I added the first company because I don't know the other and like it's mention in the text, company are encouraged to add themselves by providing a PR. At the beginning we need to start with one company but if people want to add other companies in this PR with comments, I can add them to start with more than one before merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446947444 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review Review Comment: Ok, I will update to be more specific about the issue tracker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446946324 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. Review Comment: There is an existing ASF Slack and a Kafka channel so why not using it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446946944 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. Review Comment: I will mention and add a link to StackOverFlow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
rmannibucau commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446679772 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: An alternative to reaching out (which feels like the opposite) is to add an edit link (github) and encourage people to pr to add themselves. Will make it more open and less closed to a single company so cna be a trade off maybe. Wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
rg/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Kafka is a widely used project. As such, several companies have built products and services around Kafka. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Kafka PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Kafka PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. + + + + https://www.yupiik.com; target="_blank">Yupiik contributes and commits to many Apache projects. Provides consulting, training and support Review Comment: It seems very weird to me to add this page with only a single commercial support offering listed. Honestly it feels a bit inappropriate to use the AK site to list any commercial offerings at all, but I'm not sure what the official policy is there. If we do include a commercial support section then imo we should consider reaching out to some of the other major companies offering Kafka support to ask if they would like to be included so that we provide a more complete picture of the available options. Otherwise it feels like we (the Apache Kafka project) are advocating for a particular commercial offering, which we definitely are not and should not. I still think the first part of this page could be very valuable (with the above suggestions) and would recommend splitting this PR up so that we can merge the community support section while discussing the suitability of the commercial support section -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
fpapon commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883454200 @C0urante good catch! Sorry for the copy/paste error, it's ok now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
rmannibucau commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883451784 @C0urante yes, this is a small typo @fpapon is aware of but the proposal follows ASF guidelines and the fix is trivial (before or after the merge) so a clear plus for Kafka community from the last year requests I saw, but you are right, we should have highlighted it more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
jbonofre commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446363532 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Camel is a widely used project. As such, several companies have built products and services around Camel. Review Comment: It should be Kafka here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
C0urante commented on code in PR #577: URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446362361 ## support.html: ## @@ -0,0 +1,54 @@ + + + + + + + Support + Community support + + + This is an open source project so the amount of time community members have available to help resolve your issue + is often limited as help is provided on a volunteer basis from + https://www.apache.org/foundation/how-it-works/#hats; target="_blank">individuals. + However, it is free, and you are often able to discuss issues with the developers who actually wrote the code + you are running. + + + If you want community support then please https://kafka.apache.org/contact;>contact us. + If you’re fairly certain you’re hitting a bug please report it via one of our + https://kafka.apache.org/contributing;>issue trackers. Be sure to review + these pages carefully as they contain tips and tricks about working with Apache communities in general and + Apache Kafka in particular. + + + Commercial support + + Apache Camel is a widely used project. As such, several companies have built products and services around Camel. + This page is dedicated to providing descriptions of those offerings and links to more information. + Companies are definitely encouraged to update this page directly or send a mail to the Camel PMC with a description + of your offerings, and we can update the page. The products and services listed on this page are provided for + information use only to our users. The Camel PMC does not endorse or recommend any of the products or services + on this page. See below for information about what is appropriate to add to the page. Review Comment: Wrong project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
jbonofre commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883435388 @C0urante yup it's a copy paste from Camel. I already asked @fpapon to update the page to mention Kafka instead of Camel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add get support page [kafka-site]
C0urante commented on PR #577: URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883431881 @jbonofre @rmannibucau did you actually review this PR? Part of it is copy+pasted with no changes from the [Apache Camel docs](https://github.com/apache/camel/blob/74b6e1994a33fe54c3f5a5ad2acc5849148dbb16/docs/user-manual/modules/ROOT/pages/commercial-camel-offerings.adoc#commercial-camel-offerings). I would hope that anyone who actually looked at the diff would have spotted that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add get support page [kafka-site]
fpapon opened a new pull request, #577: URL: https://github.com/apache/kafka-site/pull/577 Proposal to add a `get support` page to describe how community and commercial support is working, like we have in many other Apache projects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KAFKA-15466) Add KIP-919 support to kafka-features.sh, kafka-metadata-quorum.sh, kafka-cluster.sh
Colin McCabe created KAFKA-15466: Summary: Add KIP-919 support to kafka-features.sh, kafka-metadata-quorum.sh, kafka-cluster.sh Key: KAFKA-15466 URL: https://issues.apache.org/jira/browse/KAFKA-15466 Project: Kafka Issue Type: Improvement Reporter: Colin McCabe -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15062) Power(ppc64le) support for Kafka
Vaibhav created KAFKA-15062: --- Summary: Power(ppc64le) support for Kafka Key: KAFKA-15062 URL: https://issues.apache.org/jira/browse/KAFKA-15062 Project: Kafka Issue Type: Task Components: build Reporter: Vaibhav Support for Power architecture (ppc64le) for apache kafka. What is IBM Power architecture? It is a RISC architecture and IBM has recently made its ISA (Instruction Set Architecture) opensource and in doing so, they have significantly contributed back to the opensource community at large. Many of the pioneers of banking and HPC industries today run on ppc64le architecture. As an ongoing effort to enable open-source projects where Power architecture can add value, we are trying to enable kafka on Power. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
continue to use a transaction when
> > >> resetting
> > >> > offsets in the connector's offsets topic. Discussed in [1].
> > >> >
> > >> > 2. We would like to use a transactional ID of
> ${groupId}-${connector}
> > to
> > >> > alter/reset source connector offsets when exactly-once support is
> > >> enabled,
> > >> > where ${groupId} is the group ID of the Connect cluster and
> > >> ${connector} is
> > >> > the name of the connector. This is raised here because it would
> > >> introduce
> > >> > an additional ACL requirement for this API. A less-elegant
> alternative
> > >> that
> > >> > would obviate the additional ACL requirement is to use the
> > >> transactional ID
> > >> > that would be used by task 0 of the connector, but this may be
> > >> confusing to
> > >> > users as it could indicate that the task is actually running.
> > Discussed
> > >> in
> > >> > [2].
> > >> >
> > >> > [1] -
> > >> https://github.com/apache/kafka/pull/13465/#issuecomment-1486718538
> > >> > [2] -
> > >> https://github.com/apache/kafka/pull/13465/#discussion_r1159694956
> > >> >
> > >> > Cheers,
> > >> >
> > >> > Chris
> > >> >
> > >> > On Fri, Mar 3, 2023 at 10:22 AM Chris Egerton
> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > Thanks for the votes! I'll cast a final +1 myself and close the
> vote
> > >> out.
> > >> > >
> > >> > > This KIP passes with the following +1 votes (and no +0 or -1
> votes):
> > >> > >
> > >> > > • Greg Harris
> > >> > > • Yash Mayya
> > >> > > • Knowles Atchison Jr
> > >> > > • Mickael Maison (binding)
> > >> > > • Tom Bentley (binding)
> > >> > > • Josep Prat (binding)
> > >> > > • Chris Egerton (binding, author)
> > >> > >
> > >> > > I'll write up Jira tickets and begin implementing things next
> week.
> > >> > >
> > >> > > Cheers,
> > >> > >
> > >> > > Chris
> > >> > >
> > >> > > On Fri, Mar 3, 2023 at 10:07 AM Josep Prat
> > >>
> > >> > > wrote:
> > >> > >
> > >> > >> Hi Chris,
> > >> > >>
> > >> > >> Thanks for the KIP. I have a non-blocking comment on the DISCUSS
> > >> thread.
> > >> > >>
> > >> > >> +1 (binding).
> > >> > >>
> > >> > >> Best,
> > >> > >>
> > >> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley >
> > >> > wrote:
> > >> > >>
> > >> > >> > Hi Chris,
> > >> > >> >
> > >> > >> > Thanks for the KIP.
> > >> > >> >
> > >> > >> > +1 (binding).
> > >> > >> >
> > >> > >> > Cheers,
> > >> > >> >
> > >> > >> > Tom
> > >> > >> >
> > >> > >> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton
> > >>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Hi all,
> > >> > >> > >
> > >> > >> > > Thanks to everyone who's voted so far! Just wanted to bump
> this
> > >> > thread
> > >> > >> > and
> > >> > >> > > see if we could get a few more votes; currently we're at +3
> > >> > >> non-binding
> > >> > >> > > and +1 binding. Hoping we can get this approved, reviewed,
> and
> > >> > merged
> > >> > >> in
> > >> > >> > > time for 3.5.0.
> > >> > >> > >
> > >> > >> > > Cheers,
> > >> > >> > >
> > >> > >> > > Chris
> > >> > >> > >
> > >> > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison <
> > >> > >> mickael.mai...@gmail.com
> > >> > >> > >
> > >> > >> > > wrote:
> > >> > >> > >
> > >> > >> > > > Thanks Chris for the KIP, this is a much needed feature!
> > >> > >> > > >
> > >> > >> > > > +1 (binding)
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
> > >> > >> > > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > +1 (non binding)
> > >> > >> > > > >
> > >> > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya <
> > >> > yash.ma...@gmail.com>
> > >> > >> > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > > Hi Chris,
> > >> > >> > > > > >
> > >> > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this
> > >> > extremely
> > >> > >> > > > > > valuable addition to Kafka Connect!
> > >> > >> > > > > >
> > >> > >> > > > > > Thanks,
> > >> > >> > > > > > Yash
> > >> > >> > > > > >
> > >> > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton
> > >> > >> > > > >> > >> > > > >
> > >> > >> > > > > > wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > Hi all,
> > >> > >> > > > > > >
> > >> > >> > > > > > > I'd like to call for a vote on KIP-875, which adds
> > >> support
> > >> > for
> > >> > >> > > > viewing
> > >> > >> > > > > > and
> > >> > >> > > > > > > manipulating the offsets of connectors to the Kafka
> > >> Connect
> > >> > >> REST
> > >> > >> > > API.
> > >> > >> > > > > > >
> > >> > >> > > > > > > The KIP:
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > >> > >> > > > > > >
> > >> > >> > > > > > > The discussion thread:
> > >> > >> > > > > > >
> > >> > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
> > >> > >> > > > > > >
> > >> > >> > > > > > > Cheers,
> > >> > >> > > > > > >
> > >> > >> > > > > > > Chris
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> [image: Aiven] <https://www.aiven.io>
> > >> > >>
> > >> > >> *Josep Prat*
> > >> > >> Open Source Engineering Director, *Aiven*
> > >> > >> josep.p...@aiven.io | +491715557497
> > >> > >> aiven.io <https://www.aiven.io> | <
> > >> > >> https://www.facebook.com/aivencloud>
> > >> > >> <https://www.linkedin.com/company/aiven/> <
> > >> > >> https://twitter.com/aiven_io>
> > >> > >> *Aiven Deutschland GmbH*
> > >> > >> Alexanderufer 3-7, 10117 Berlin
> > >> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
al ACL requirement is to use the > >> transactional ID > >> > that would be used by task 0 of the connector, but this may be > >> confusing to > >> > users as it could indicate that the task is actually running. > Discussed > >> in > >> > [2]. > >> > > >> > [1] - > >> https://github.com/apache/kafka/pull/13465/#issuecomment-1486718538 > >> > [2] - > >> https://github.com/apache/kafka/pull/13465/#discussion_r1159694956 > >> > > >> > Cheers, > >> > > >> > Chris > >> > > >> > On Fri, Mar 3, 2023 at 10:22 AM Chris Egerton > wrote: > >> > > >> > > Hi all, > >> > > > >> > > Thanks for the votes! I'll cast a final +1 myself and close the vote > >> out. > >> > > > >> > > This KIP passes with the following +1 votes (and no +0 or -1 votes): > >> > > > >> > > • Greg Harris > >> > > • Yash Mayya > >> > > • Knowles Atchison Jr > >> > > • Mickael Maison (binding) > >> > > • Tom Bentley (binding) > >> > > • Josep Prat (binding) > >> > > • Chris Egerton (binding, author) > >> > > > >> > > I'll write up Jira tickets and begin implementing things next week. > >> > > > >> > > Cheers, > >> > > > >> > > Chris > >> > > > >> > > On Fri, Mar 3, 2023 at 10:07 AM Josep Prat > >> > >> > > wrote: > >> > > > >> > >> Hi Chris, > >> > >> > >> > >> Thanks for the KIP. I have a non-blocking comment on the DISCUSS > >> thread. > >> > >> > >> > >> +1 (binding). > >> > >> > >> > >> Best, > >> > >> > >> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley > >> > wrote: > >> > >> > >> > >> > Hi Chris, > >> > >> > > >> > >> > Thanks for the KIP. > >> > >> > > >> > >> > +1 (binding). > >> > >> > > >> > >> > Cheers, > >> > >> > > >> > >> > Tom > >> > >> > > >> > >> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton > >> > >> > >> > wrote: > >> > >> > > >> > >> > > Hi all, > >> > >> > > > >> > >> > > Thanks to everyone who's voted so far! Just wanted to bump this > >> > thread > >> > >> > and > >> > >> > > see if we could get a few more votes; currently we're at +3 > >> > >> non-binding > >> > >> > > and +1 binding. Hoping we can get this approved, reviewed, and > >> > merged > >> > >> in > >> > >> > > time for 3.5.0. > >> > >> > > > >> > >> > > Cheers, > >> > >> > > > >> > >> > > Chris > >> > >> > > > >> > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison < > >> > >> mickael.mai...@gmail.com > >> > >> > > > >> > >> > > wrote: > >> > >> > > > >> > >> > > > Thanks Chris for the KIP, this is a much needed feature! > >> > >> > > > > >> > >> > > > +1 (binding) > >> > >> > > > > >> > >> > > > > >> > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > >> > >> > > > wrote: > >> > >> > > > > > >> > >> > > > > +1 (non binding) > >> > >> > > > > > >> > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya < > >> > yash.ma...@gmail.com> > >> > >> > > wrote: > >> > >> > > > > > >> > >> > > > > > Hi Chris, > >> > >> > > > > > > >> > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this > >> > extremely > >> > >> > > > > > valuable addition to Kafka Connect! > >> > >> > > > > > > >> > >> > > > > > Thanks, > >> > >> > > > > > Yash > >> > >> > > > > > > >> > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > >> > >> > > >> > >> > > > > > >> > >> > > > > > wrote: > >> > >> > > > > > > >> > >> > > > > > > Hi all, > >> > >> > > > > > > > >> > >> > > > > > > I'd like to call for a vote on KIP-875, which adds > >> support > >> > for > >> > >> > > > viewing > >> > >> > > > > > and > >> > >> > > > > > > manipulating the offsets of connectors to the Kafka > >> Connect > >> > >> REST > >> > >> > > API. > >> > >> > > > > > > > >> > >> > > > > > > The KIP: > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > >> > >> > > > > > > > >> > >> > > > > > > The discussion thread: > >> > >> > > > > > > > >> > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > >> > >> > > > > > > > >> > >> > > > > > > Cheers, > >> > >> > > > > > > > >> > >> > > > > > > Chris > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> > >> > >> -- > >> > >> [image: Aiven] <https://www.aiven.io> > >> > >> > >> > >> *Josep Prat* > >> > >> Open Source Engineering Director, *Aiven* > >> > >> josep.p...@aiven.io | +491715557497 > >> > >> aiven.io <https://www.aiven.io> | < > >> > >> https://www.facebook.com/aivencloud> > >> > >> <https://www.linkedin.com/company/aiven/> < > >> > >> https://twitter.com/aiven_io> > >> > >> *Aiven Deutschland GmbH* > >> > >> Alexanderufer 3-7, 10117 Berlin > >> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > >> > >> Amtsgericht Charlottenburg, HRB 209739 B > >> > >> > >> > > > >> > > >> > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
eers, >> > > >> > > Chris >> > > >> > > On Fri, Mar 3, 2023 at 10:07 AM Josep Prat >> >> > > wrote: >> > > >> > >> Hi Chris, >> > >> >> > >> Thanks for the KIP. I have a non-blocking comment on the DISCUSS >> thread. >> > >> >> > >> +1 (binding). >> > >> >> > >> Best, >> > >> >> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley >> > wrote: >> > >> >> > >> > Hi Chris, >> > >> > >> > >> > Thanks for the KIP. >> > >> > >> > >> > +1 (binding). >> > >> > >> > >> > Cheers, >> > >> > >> > >> > Tom >> > >> > >> > >> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton >> >> > >> > wrote: >> > >> > >> > >> > > Hi all, >> > >> > > >> > >> > > Thanks to everyone who's voted so far! Just wanted to bump this >> > thread >> > >> > and >> > >> > > see if we could get a few more votes; currently we're at +3 >> > >> non-binding >> > >> > > and +1 binding. Hoping we can get this approved, reviewed, and >> > merged >> > >> in >> > >> > > time for 3.5.0. >> > >> > > >> > >> > > Cheers, >> > >> > > >> > >> > > Chris >> > >> > > >> > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison < >> > >> mickael.mai...@gmail.com >> > >> > > >> > >> > > wrote: >> > >> > > >> > >> > > > Thanks Chris for the KIP, this is a much needed feature! >> > >> > > > >> > >> > > > +1 (binding) >> > >> > > > >> > >> > > > >> > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr >> > >> > > > wrote: >> > >> > > > > >> > >> > > > > +1 (non binding) >> > >> > > > > >> > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya < >> > yash.ma...@gmail.com> >> > >> > > wrote: >> > >> > > > > >> > >> > > > > > Hi Chris, >> > >> > > > > > >> > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this >> > extremely >> > >> > > > > > valuable addition to Kafka Connect! >> > >> > > > > > >> > >> > > > > > Thanks, >> > >> > > > > > Yash >> > >> > > > > > >> > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton >> > >> > > > > >> > > > > >> > >> > > > > > wrote: >> > >> > > > > > >> > >> > > > > > > Hi all, >> > >> > > > > > > >> > >> > > > > > > I'd like to call for a vote on KIP-875, which adds >> support >> > for >> > >> > > > viewing >> > >> > > > > > and >> > >> > > > > > > manipulating the offsets of connectors to the Kafka >> Connect >> > >> REST >> > >> > > API. >> > >> > > > > > > >> > >> > > > > > > The KIP: >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect >> > >> > > > > > > >> > >> > > > > > > The discussion thread: >> > >> > > > > > > >> > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 >> > >> > > > > > > >> > >> > > > > > > Cheers, >> > >> > > > > > > >> > >> > > > > > > Chris >> > >> > > > > > > >> > >> > > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> >> > >> -- >> > >> [image: Aiven] <https://www.aiven.io> >> > >> >> > >> *Josep Prat* >> > >> Open Source Engineering Director, *Aiven* >> > >> josep.p...@aiven.io | +491715557497 >> > >> aiven.io <https://www.aiven.io> | < >> > >> https://www.facebook.com/aivencloud> >> > >> <https://www.linkedin.com/company/aiven/> < >> > >> https://twitter.com/aiven_io> >> > >> *Aiven Deutschland GmbH* >> > >> Alexanderufer 3-7, 10117 Berlin >> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > >> Amtsgericht Charlottenburg, HRB 209739 B >> > >> >> > > >> > >> >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
16:11, Chris Egerton > > > >> > wrote: > > >> > > > >> > > Hi all, > > >> > > > > >> > > Thanks to everyone who's voted so far! Just wanted to bump this > > thread > > >> > and > > >> > > see if we could get a few more votes; currently we're at +3 > > >> non-binding > > >> > > and +1 binding. Hoping we can get this approved, reviewed, and > > merged > > >> in > > >> > > time for 3.5.0. > > >> > > > > >> > > Cheers, > > >> > > > > >> > > Chris > > >> > > > > >> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison < > > >> mickael.mai...@gmail.com > > >> > > > > >> > > wrote: > > >> > > > > >> > > > Thanks Chris for the KIP, this is a much needed feature! > > >> > > > > > >> > > > +1 (binding) > > >> > > > > > >> > > > > > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > > >> > > > wrote: > > >> > > > > > > >> > > > > +1 (non binding) > > >> > > > > > > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya < > > yash.ma...@gmail.com> > > >> > > wrote: > > >> > > > > > > >> > > > > > Hi Chris, > > >> > > > > > > > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this > > extremely > > >> > > > > > valuable addition to Kafka Connect! > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Yash > > >> > > > > > > > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > >> > > > >> > > > > > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Hi all, > > >> > > > > > > > > >> > > > > > > I'd like to call for a vote on KIP-875, which adds support > > for > > >> > > > viewing > > >> > > > > > and > > >> > > > > > > manipulating the offsets of connectors to the Kafka > Connect > > >> REST > > >> > > API. > > >> > > > > > > > > >> > > > > > > The KIP: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > >> > > > > > > > > >> > > > > > > The discussion thread: > > >> > > > > > > > > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > >> > > > > > > > > >> > > > > > > Cheers, > > >> > > > > > > > > >> > > > > > > Chris > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> -- > > >> [image: Aiven] <https://www.aiven.io> > > >> > > >> *Josep Prat* > > >> Open Source Engineering Director, *Aiven* > > >> josep.p...@aiven.io | +491715557497 > > >> aiven.io <https://www.aiven.io> | < > > >> https://www.facebook.com/aivencloud> > > >> <https://www.linkedin.com/company/aiven/> < > > >> https://twitter.com/aiven_io> > > >> *Aiven Deutschland GmbH* > > >> Alexanderufer 3-7, 10117 Berlin > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
t; >> > > > Thanks Chris for the KIP, this is a much needed feature! > >> > > > > >> > > > +1 (binding) > >> > > > > >> > > > > >> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > >> > > > wrote: > >> > > > > > >> > > > > +1 (non binding) > >> > > > > > >> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya < > yash.ma...@gmail.com> > >> > > wrote: > >> > > > > > >> > > > > > Hi Chris, > >> > > > > > > >> > > > > > I'm +1 (non-binding). Thanks again for proposing this > extremely > >> > > > > > valuable addition to Kafka Connect! > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Yash > >> > > > > > > >> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > >> > > >> > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi all, > >> > > > > > > > >> > > > > > > I'd like to call for a vote on KIP-875, which adds support > for > >> > > > viewing > >> > > > > > and > >> > > > > > > manipulating the offsets of connectors to the Kafka Connect > >> REST > >> > > API. > >> > > > > > > > >> > > > > > > The KIP: > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > >> > > > > > > > >> > > > > > > The discussion thread: > >> > > > > > > > >> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > >> > > > > > > > >> > > > > > > Cheers, > >> > > > > > > > >> > > > > > > Chris > >> > > > > > > > >> > > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> [image: Aiven] <https://www.aiven.io> > >> > >> *Josep Prat* > >> Open Source Engineering Director, *Aiven* > >> josep.p...@aiven.io | +491715557497 > >> aiven.io <https://www.aiven.io> | < > >> https://www.facebook.com/aivencloud> > >> <https://www.linkedin.com/company/aiven/> < > >> https://twitter.com/aiven_io> > >> *Aiven Deutschland GmbH* > >> Alexanderufer 3-7, 10117 Berlin > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > >> Amtsgericht Charlottenburg, HRB 209739 B > >> > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi all,
A couple slight tweaks to the design have been proposed during
implementation and I'd like to report them here to make sure that they're
acceptable to all who previously voted for this KIP. I've updated the KIP
to include these changes but will be happy to revert and/or amend if there
are any concerns.
1. We would like to refrain from using a transaction when resetting source
connector offsets in the worker's global offsets topic when exactly-once
support is enabled. We would continue to use a transaction when resetting
offsets in the connector's offsets topic. Discussed in [1].
2. We would like to use a transactional ID of ${groupId}-${connector} to
alter/reset source connector offsets when exactly-once support is enabled,
where ${groupId} is the group ID of the Connect cluster and ${connector} is
the name of the connector. This is raised here because it would introduce
an additional ACL requirement for this API. A less-elegant alternative that
would obviate the additional ACL requirement is to use the transactional ID
that would be used by task 0 of the connector, but this may be confusing to
users as it could indicate that the task is actually running. Discussed in
[2].
[1] - https://github.com/apache/kafka/pull/13465/#issuecomment-1486718538
[2] - https://github.com/apache/kafka/pull/13465/#discussion_r1159694956
Cheers,
Chris
On Fri, Mar 3, 2023 at 10:22 AM Chris Egerton wrote:
> Hi all,
>
> Thanks for the votes! I'll cast a final +1 myself and close the vote out.
>
> This KIP passes with the following +1 votes (and no +0 or -1 votes):
>
> • Greg Harris
> • Yash Mayya
> • Knowles Atchison Jr
> • Mickael Maison (binding)
> • Tom Bentley (binding)
> • Josep Prat (binding)
> • Chris Egerton (binding, author)
>
> I'll write up Jira tickets and begin implementing things next week.
>
> Cheers,
>
> Chris
>
> On Fri, Mar 3, 2023 at 10:07 AM Josep Prat
> wrote:
>
>> Hi Chris,
>>
>> Thanks for the KIP. I have a non-blocking comment on the DISCUSS thread.
>>
>> +1 (binding).
>>
>> Best,
>>
>> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley wrote:
>>
>> > Hi Chris,
>> >
>> > Thanks for the KIP.
>> >
>> > +1 (binding).
>> >
>> > Cheers,
>> >
>> > Tom
>> >
>> > On Wed, 15 Feb 2023 at 16:11, Chris Egerton
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > Thanks to everyone who's voted so far! Just wanted to bump this thread
>> > and
>> > > see if we could get a few more votes; currently we're at +3
>> non-binding
>> > > and +1 binding. Hoping we can get this approved, reviewed, and merged
>> in
>> > > time for 3.5.0.
>> > >
>> > > Cheers,
>> > >
>> > > Chris
>> > >
>> > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison <
>> mickael.mai...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Thanks Chris for the KIP, this is a much needed feature!
>> > > >
>> > > > +1 (binding)
>> > > >
>> > > >
>> > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
>> > > > wrote:
>> > > > >
>> > > > > +1 (non binding)
>> > > > >
>> > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya
>> > > wrote:
>> > > > >
>> > > > > > Hi Chris,
>> > > > > >
>> > > > > > I'm +1 (non-binding). Thanks again for proposing this extremely
>> > > > > > valuable addition to Kafka Connect!
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Yash
>> > > > > >
>> > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton
>> > > > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > I'd like to call for a vote on KIP-875, which adds support for
>> > > > viewing
>> > > > > > and
>> > > > > > > manipulating the offsets of connectors to the Kafka Connect
>> REST
>> > > API.
>> > > > > > >
>> > > > > > > The KIP:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
>> > > > > > >
>> > > > > > > The discussion thread:
>> > > > > > >
>> https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > >
>> > > > > > > Chris
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> [image: Aiven] <https://www.aiven.io>
>>
>> *Josep Prat*
>> Open Source Engineering Director, *Aiven*
>> josep.p...@aiven.io | +491715557497
>> aiven.io <https://www.aiven.io> | <
>> https://www.facebook.com/aivencloud>
>> <https://www.linkedin.com/company/aiven/> <
>> https://twitter.com/aiven_io>
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi all, Thanks for the votes! I'll cast a final +1 myself and close the vote out. This KIP passes with the following +1 votes (and no +0 or -1 votes): • Greg Harris • Yash Mayya • Knowles Atchison Jr • Mickael Maison (binding) • Tom Bentley (binding) • Josep Prat (binding) • Chris Egerton (binding, author) I'll write up Jira tickets and begin implementing things next week. Cheers, Chris On Fri, Mar 3, 2023 at 10:07 AM Josep Prat wrote: > Hi Chris, > > Thanks for the KIP. I have a non-blocking comment on the DISCUSS thread. > > +1 (binding). > > Best, > > On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley wrote: > > > Hi Chris, > > > > Thanks for the KIP. > > > > +1 (binding). > > > > Cheers, > > > > Tom > > > > On Wed, 15 Feb 2023 at 16:11, Chris Egerton > > wrote: > > > > > Hi all, > > > > > > Thanks to everyone who's voted so far! Just wanted to bump this thread > > and > > > see if we could get a few more votes; currently we're at +3 non-binding > > > and +1 binding. Hoping we can get this approved, reviewed, and merged > in > > > time for 3.5.0. > > > > > > Cheers, > > > > > > Chris > > > > > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison < > mickael.mai...@gmail.com > > > > > > wrote: > > > > > > > Thanks Chris for the KIP, this is a much needed feature! > > > > > > > > +1 (binding) > > > > > > > > > > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > > > > wrote: > > > > > > > > > > +1 (non binding) > > > > > > > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya > > > wrote: > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > I'm +1 (non-binding). Thanks again for proposing this extremely > > > > > > valuable addition to Kafka Connect! > > > > > > > > > > > > Thanks, > > > > > > Yash > > > > > > > > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to call for a vote on KIP-875, which adds support for > > > > viewing > > > > > > and > > > > > > > manipulating the offsets of connectors to the Kafka Connect > REST > > > API. > > > > > > > > > > > > > > The KIP: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > > > > > > > The discussion thread: > > > > > > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > > > > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud > > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, Thanks for the KIP. I have a non-blocking comment on the DISCUSS thread. +1 (binding). Best, On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley wrote: > Hi Chris, > > Thanks for the KIP. > > +1 (binding). > > Cheers, > > Tom > > On Wed, 15 Feb 2023 at 16:11, Chris Egerton > wrote: > > > Hi all, > > > > Thanks to everyone who's voted so far! Just wanted to bump this thread > and > > see if we could get a few more votes; currently we're at +3 non-binding > > and +1 binding. Hoping we can get this approved, reviewed, and merged in > > time for 3.5.0. > > > > Cheers, > > > > Chris > > > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison > > > wrote: > > > > > Thanks Chris for the KIP, this is a much needed feature! > > > > > > +1 (binding) > > > > > > > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > > > wrote: > > > > > > > > +1 (non binding) > > > > > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya > > wrote: > > > > > > > > > Hi Chris, > > > > > > > > > > I'm +1 (non-binding). Thanks again for proposing this extremely > > > > > valuable addition to Kafka Connect! > > > > > > > > > > Thanks, > > > > > Yash > > > > > > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > > > > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like to call for a vote on KIP-875, which adds support for > > > viewing > > > > > and > > > > > > manipulating the offsets of connectors to the Kafka Connect REST > > API. > > > > > > > > > > > > The KIP: > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > > > > > The discussion thread: > > > > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, Thanks for the KIP. +1 (binding). Cheers, Tom On Wed, 15 Feb 2023 at 16:11, Chris Egerton wrote: > Hi all, > > Thanks to everyone who's voted so far! Just wanted to bump this thread and > see if we could get a few more votes; currently we're at +3 non-binding > and +1 binding. Hoping we can get this approved, reviewed, and merged in > time for 3.5.0. > > Cheers, > > Chris > > On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison > wrote: > > > Thanks Chris for the KIP, this is a much needed feature! > > > > +1 (binding) > > > > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > > wrote: > > > > > > +1 (non binding) > > > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya > wrote: > > > > > > > Hi Chris, > > > > > > > > I'm +1 (non-binding). Thanks again for proposing this extremely > > > > valuable addition to Kafka Connect! > > > > > > > > Thanks, > > > > Yash > > > > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to call for a vote on KIP-875, which adds support for > > viewing > > > > and > > > > > manipulating the offsets of connectors to the Kafka Connect REST > API. > > > > > > > > > > The KIP: > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > > > The discussion thread: > > > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > > > > > > > Cheers, > > > > > > > > > > Chris > > > > > > > > > > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi all, Thanks to everyone who's voted so far! Just wanted to bump this thread and see if we could get a few more votes; currently we're at +3 non-binding and +1 binding. Hoping we can get this approved, reviewed, and merged in time for 3.5.0. Cheers, Chris On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison wrote: > Thanks Chris for the KIP, this is a much needed feature! > > +1 (binding) > > > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr > wrote: > > > > +1 (non binding) > > > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya wrote: > > > > > Hi Chris, > > > > > > I'm +1 (non-binding). Thanks again for proposing this extremely > > > valuable addition to Kafka Connect! > > > > > > Thanks, > > > Yash > > > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to call for a vote on KIP-875, which adds support for > viewing > > > and > > > > manipulating the offsets of connectors to the Kafka Connect REST API. > > > > > > > > The KIP: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > The discussion thread: > > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Thanks Chris for the KIP, this is a much needed feature! +1 (binding) On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr wrote: > > +1 (non binding) > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya wrote: > > > Hi Chris, > > > > I'm +1 (non-binding). Thanks again for proposing this extremely > > valuable addition to Kafka Connect! > > > > Thanks, > > Yash > > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > > wrote: > > > > > Hi all, > > > > > > I'd like to call for a vote on KIP-875, which adds support for viewing > > and > > > manipulating the offsets of connectors to the Kafka Connect REST API. > > > > > > The KIP: > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > The discussion thread: > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > > > Cheers, > > > > > > Chris > > > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
+1 (non binding) On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya wrote: > Hi Chris, > > I'm +1 (non-binding). Thanks again for proposing this extremely > valuable addition to Kafka Connect! > > Thanks, > Yash > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton > wrote: > > > Hi all, > > > > I'd like to call for a vote on KIP-875, which adds support for viewing > and > > manipulating the offsets of connectors to the Kafka Connect REST API. > > > > The KIP: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > The discussion thread: > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > > > Cheers, > > > > Chris > > >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, I'm +1 (non-binding). Thanks again for proposing this extremely valuable addition to Kafka Connect! Thanks, Yash On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton wrote: > Hi all, > > I'd like to call for a vote on KIP-875, which adds support for viewing and > manipulating the offsets of connectors to the Kafka Connect REST API. > > The KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > The discussion thread: > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > Cheers, > > Chris >
Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, I'm +1 (non-binding) for KIP-875. Thanks for proposing this change! Greg Harris On Wed, Jan 18, 2023 at 10:41 AM Chris Egerton wrote: > Hi all, > > I'd like to call for a vote on KIP-875, which adds support for viewing and > manipulating the offsets of connectors to the Kafka Connect REST API. > > The KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > The discussion thread: > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 > > Cheers, > > Chris >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
he Stopped state to add an > exception for temporary starts, while still preventing it from using > resources in the background. > We should also consider whether we can influence the rebalance algorithm to > allocate STOPPED connectors to the leader, or not allocate them at all, or > use the rebalance algorithm's connector assignment to distribute the > alterOffsets calls across the cluster. > > And slightly related: > 3. How will the check for the STOPPED state on alter requests be > implemented, is it from reading the config topic or the status topic? > 4. Is there synchronization to ensure that a connector on a non-leader is > STOPPED before an instance is started on the leader? If not, there might be > a risk of the non-leader connector overwriting the effects of the > alterOffsets on the leader connector. > > Thanks, > Greg > > > On Fri, Dec 16, 2022 at 8:26 AM Yash Mayya wrote: > > > Hi Chris, > > > > Thanks for clarifying, I had missed that update in the KIP (the bit about > > altering/resetting offsets response). I think your arguments for not > going > > with an additional method or a custom return type make sense. > > > > Thanks, > > Yash > > > > On Sat, Dec 10, 2022 at 12:28 AM Chris Egerton > > wrote: > > > > > Hi Yash, > > > > > > The idea with the boolean is to just signify that a connector has > > > overridden this method, which allows us to issue a definitive response > in > > > the REST API when servicing offset alter/reset requests (described more > > in > > > detail here: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Altering/resettingoffsets(response) > > > ). > > > > > > One alternative could be to add some kind of OffsetAlterResult return > > type > > > for this method with constructors like "OffsetAlterResult.success()", > > > "OffsetAlterResult.unknown()" (default), and > > > "OffsetAlterResult.failure(Throwable t)", but it's hard to envision a > > > reason to use the "failure" static factory method instead of just > > throwing > > > an exception, at which point, we're only left with two reasonable > > methods: > > > success, and unknown. > > > > > > Another could be to add a "boolean canResetOffsets()" method to the > > > SourceConnector class, but adding a separate method seems like > overkill, > > > and developers may not understand that implementing that method to > always > > > return "false" won't actually cause offset reset requests to not take > > > place. > > > > > > One final option could be to have something like an AlterOffsetsSupport > > > enum with values SUPPORTED and UNSUPPORTED and a new > "AlterOffsetsSupport > > > alterOffsetsSupport()" SourceConnector method that returns null by > > default > > > (which implicitly maps to the "unknown support" response message in the > > > REST API). This would line up with the ExactlyOnceSupport API we added > in > > > KIP-618. However, I'm hesitant to adopt it because it's not strictly > > > necessary to address the cases that we want to cover; everything can be > > > handled with a single method that gets invoked on offset alter/reset > > > requests. With exactly-once support, we added this hook because it was > > > designed to be invoked in a different point in the connector lifecycle. > > > > > > Cheers, > > > > > > Chris > > > > > > On Wed, Dec 7, 2022 at 9:46 AM Yash Mayya > wrote: > > > > > > > Hi Chris, > > > > > > > > Sorry for the late reply. > > > > > > > > > I don't believe logging an error message is sufficient for > > > > > handling failures to reset-after-delete. IMO it's highly > > > > > likely that users will either shoot themselves in the foot > > > > > by not reading the fine print and realizing that the offset > > > > > request may have failed, or will ask for better visibility > > > > > into the success or failure of the reset request than > > > > > scanning log files. > > > > > > > > Your reasoning for deferring the reset offsets after delete > > functionality > > > > to a separate KIP makes sense, thanks for the explanation. > > > > > > > > > I'v
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, I had some clarifying questions about the alterOffsets hooks. The KIP includes these elements of the design: * The Javadoc for the methods mentions that the alterOffsets methods are only called on started and initialized connector objects. * The 'Altering' and 'Resetting' offsets descriptions indicate that the requests are forwarded to the leader. * And finally, the description of "A new STOPPED state" includes a note that the Connector will not be started, and it will not be able to generate new task configurations 1. Does this mean that a connector may be assigned to a non-leader worker in the cluster, an alter request comes in, and a connector instance is temporarily started on the leader to service the request? 2. While the alterOffsets method is being called, will the leader need to ignore potential requestTaskReconfiguration calls? On the surface, this seems to conflict with the semantics of the rebalance subsystem, as connectors are being started where they are not assigned. Additionally, it seems to conflict with the semantics of the STOPPED state, which when read literally, might imply that the connector is not started _anywhere_ in the cluster. I think that if we wish to provide these alterOffsets methods, they must be called on started and initialized connector objects. And if that's the case, then we will need to ignore requestTaskReconfigurationCalls. But we may need to relax the wording on the Stopped state to add an exception for temporary starts, while still preventing it from using resources in the background. We should also consider whether we can influence the rebalance algorithm to allocate STOPPED connectors to the leader, or not allocate them at all, or use the rebalance algorithm's connector assignment to distribute the alterOffsets calls across the cluster. And slightly related: 3. How will the check for the STOPPED state on alter requests be implemented, is it from reading the config topic or the status topic? 4. Is there synchronization to ensure that a connector on a non-leader is STOPPED before an instance is started on the leader? If not, there might be a risk of the non-leader connector overwriting the effects of the alterOffsets on the leader connector. Thanks, Greg On Fri, Dec 16, 2022 at 8:26 AM Yash Mayya wrote: > Hi Chris, > > Thanks for clarifying, I had missed that update in the KIP (the bit about > altering/resetting offsets response). I think your arguments for not going > with an additional method or a custom return type make sense. > > Thanks, > Yash > > On Sat, Dec 10, 2022 at 12:28 AM Chris Egerton > wrote: > > > Hi Yash, > > > > The idea with the boolean is to just signify that a connector has > > overridden this method, which allows us to issue a definitive response in > > the REST API when servicing offset alter/reset requests (described more > in > > detail here: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Altering/resettingoffsets(response) > > ). > > > > One alternative could be to add some kind of OffsetAlterResult return > type > > for this method with constructors like "OffsetAlterResult.success()", > > "OffsetAlterResult.unknown()" (default), and > > "OffsetAlterResult.failure(Throwable t)", but it's hard to envision a > > reason to use the "failure" static factory method instead of just > throwing > > an exception, at which point, we're only left with two reasonable > methods: > > success, and unknown. > > > > Another could be to add a "boolean canResetOffsets()" method to the > > SourceConnector class, but adding a separate method seems like overkill, > > and developers may not understand that implementing that method to always > > return "false" won't actually cause offset reset requests to not take > > place. > > > > One final option could be to have something like an AlterOffsetsSupport > > enum with values SUPPORTED and UNSUPPORTED and a new "AlterOffsetsSupport > > alterOffsetsSupport()" SourceConnector method that returns null by > default > > (which implicitly maps to the "unknown support" response message in the > > REST API). This would line up with the ExactlyOnceSupport API we added in > > KIP-618. However, I'm hesitant to adopt it because it's not strictly > > necessary to address the cases that we want to cover; everything can be > > handled with a single method that gets invoked on offset alter/reset > > requests. With exactly-once support, we added this hook because it was > > designed to be invoked in a different point in the connector lifecycle. &g
[VOTE] KIP-875: First-class offsets support in Kafka Connect
Hi all, I'd like to call for a vote on KIP-875, which adds support for viewing and manipulating the offsets of connectors to the Kafka Connect REST API. The KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect The discussion thread: https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02 Cheers, Chris
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, Thanks for clarifying, I had missed that update in the KIP (the bit about altering/resetting offsets response). I think your arguments for not going with an additional method or a custom return type make sense. Thanks, Yash On Sat, Dec 10, 2022 at 12:28 AM Chris Egerton wrote: > Hi Yash, > > The idea with the boolean is to just signify that a connector has > overridden this method, which allows us to issue a definitive response in > the REST API when servicing offset alter/reset requests (described more in > detail here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Altering/resettingoffsets(response) > ). > > One alternative could be to add some kind of OffsetAlterResult return type > for this method with constructors like "OffsetAlterResult.success()", > "OffsetAlterResult.unknown()" (default), and > "OffsetAlterResult.failure(Throwable t)", but it's hard to envision a > reason to use the "failure" static factory method instead of just throwing > an exception, at which point, we're only left with two reasonable methods: > success, and unknown. > > Another could be to add a "boolean canResetOffsets()" method to the > SourceConnector class, but adding a separate method seems like overkill, > and developers may not understand that implementing that method to always > return "false" won't actually cause offset reset requests to not take > place. > > One final option could be to have something like an AlterOffsetsSupport > enum with values SUPPORTED and UNSUPPORTED and a new "AlterOffsetsSupport > alterOffsetsSupport()" SourceConnector method that returns null by default > (which implicitly maps to the "unknown support" response message in the > REST API). This would line up with the ExactlyOnceSupport API we added in > KIP-618. However, I'm hesitant to adopt it because it's not strictly > necessary to address the cases that we want to cover; everything can be > handled with a single method that gets invoked on offset alter/reset > requests. With exactly-once support, we added this hook because it was > designed to be invoked in a different point in the connector lifecycle. > > Cheers, > > Chris > > On Wed, Dec 7, 2022 at 9:46 AM Yash Mayya wrote: > > > Hi Chris, > > > > Sorry for the late reply. > > > > > I don't believe logging an error message is sufficient for > > > handling failures to reset-after-delete. IMO it's highly > > > likely that users will either shoot themselves in the foot > > > by not reading the fine print and realizing that the offset > > > request may have failed, or will ask for better visibility > > > into the success or failure of the reset request than > > > scanning log files. > > > > Your reasoning for deferring the reset offsets after delete functionality > > to a separate KIP makes sense, thanks for the explanation. > > > > > I've updated the KIP with the > > > developer-facing API changes for this logic > > > > This is great, I hadn't considered the other two (very valid) use-cases > for > > such an API, thanks for adding these with elaborate documentation! > However, > > the significance / use of the boolean value returned by the two methods > is > > not fully clear, could you please clarify? > > > > Thanks, > > Yash > > > > On Fri, Nov 18, 2022 at 1:06 AM Chris Egerton > > wrote: > > > > > Hi Yash, > > > > > > I've updated the KIP with the correct "kafka_topic", "kafka_partition", > > and > > > "kafka_offset" keys in the JSON examples (settled on those instead of > > > prefixing with "Kafka " for better interactions with tooling like JQ). > > I've > > > also added a note about sink offset requests failing if there are still > > > active members in the consumer group. > > > > > > I don't believe logging an error message is sufficient for handling > > > failures to reset-after-delete. IMO it's highly likely that users will > > > either shoot themselves in the foot by not reading the fine print and > > > realizing that the offset request may have failed, or will ask for > better > > > visibility into the success or failure of the reset request than > scanning > > > log files. I don't doubt that there are ways to address this, but I > would > > > prefer to leave them to a separate KIP since the required design work > is > > > non-trivial and I
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Yash, The idea with the boolean is to just signify that a connector has overridden this method, which allows us to issue a definitive response in the REST API when servicing offset alter/reset requests (described more in detail here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Altering/resettingoffsets(response) ). One alternative could be to add some kind of OffsetAlterResult return type for this method with constructors like "OffsetAlterResult.success()", "OffsetAlterResult.unknown()" (default), and "OffsetAlterResult.failure(Throwable t)", but it's hard to envision a reason to use the "failure" static factory method instead of just throwing an exception, at which point, we're only left with two reasonable methods: success, and unknown. Another could be to add a "boolean canResetOffsets()" method to the SourceConnector class, but adding a separate method seems like overkill, and developers may not understand that implementing that method to always return "false" won't actually cause offset reset requests to not take place. One final option could be to have something like an AlterOffsetsSupport enum with values SUPPORTED and UNSUPPORTED and a new "AlterOffsetsSupport alterOffsetsSupport()" SourceConnector method that returns null by default (which implicitly maps to the "unknown support" response message in the REST API). This would line up with the ExactlyOnceSupport API we added in KIP-618. However, I'm hesitant to adopt it because it's not strictly necessary to address the cases that we want to cover; everything can be handled with a single method that gets invoked on offset alter/reset requests. With exactly-once support, we added this hook because it was designed to be invoked in a different point in the connector lifecycle. Cheers, Chris On Wed, Dec 7, 2022 at 9:46 AM Yash Mayya wrote: > Hi Chris, > > Sorry for the late reply. > > > I don't believe logging an error message is sufficient for > > handling failures to reset-after-delete. IMO it's highly > > likely that users will either shoot themselves in the foot > > by not reading the fine print and realizing that the offset > > request may have failed, or will ask for better visibility > > into the success or failure of the reset request than > > scanning log files. > > Your reasoning for deferring the reset offsets after delete functionality > to a separate KIP makes sense, thanks for the explanation. > > > I've updated the KIP with the > > developer-facing API changes for this logic > > This is great, I hadn't considered the other two (very valid) use-cases for > such an API, thanks for adding these with elaborate documentation! However, > the significance / use of the boolean value returned by the two methods is > not fully clear, could you please clarify? > > Thanks, > Yash > > On Fri, Nov 18, 2022 at 1:06 AM Chris Egerton > wrote: > > > Hi Yash, > > > > I've updated the KIP with the correct "kafka_topic", "kafka_partition", > and > > "kafka_offset" keys in the JSON examples (settled on those instead of > > prefixing with "Kafka " for better interactions with tooling like JQ). > I've > > also added a note about sink offset requests failing if there are still > > active members in the consumer group. > > > > I don't believe logging an error message is sufficient for handling > > failures to reset-after-delete. IMO it's highly likely that users will > > either shoot themselves in the foot by not reading the fine print and > > realizing that the offset request may have failed, or will ask for better > > visibility into the success or failure of the reset request than scanning > > log files. I don't doubt that there are ways to address this, but I would > > prefer to leave them to a separate KIP since the required design work is > > non-trivial and I do not feel that the added burden is worth tying to > this > > KIP as a blocker. > > > > I was really hoping to avoid introducing a change to the developer-facing > > APIs with this KIP, but after giving it some thought I think this may be > > unavoidable. It's debatable whether validation of altered offsets is a > good > > enough use case on its own for this kind of API, but since there are also > > connectors out there that manage offsets externally, we should probably > add > > a hook to allow those external offsets to be managed, which can then > serve > > double- or even-triple duty as a hook to validate custom offsets and to > > notify users whether offset resets/a
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, Sorry for the late reply. > I don't believe logging an error message is sufficient for > handling failures to reset-after-delete. IMO it's highly > likely that users will either shoot themselves in the foot > by not reading the fine print and realizing that the offset > request may have failed, or will ask for better visibility > into the success or failure of the reset request than > scanning log files. Your reasoning for deferring the reset offsets after delete functionality to a separate KIP makes sense, thanks for the explanation. > I've updated the KIP with the > developer-facing API changes for this logic This is great, I hadn't considered the other two (very valid) use-cases for such an API, thanks for adding these with elaborate documentation! However, the significance / use of the boolean value returned by the two methods is not fully clear, could you please clarify? Thanks, Yash On Fri, Nov 18, 2022 at 1:06 AM Chris Egerton wrote: > Hi Yash, > > I've updated the KIP with the correct "kafka_topic", "kafka_partition", and > "kafka_offset" keys in the JSON examples (settled on those instead of > prefixing with "Kafka " for better interactions with tooling like JQ). I've > also added a note about sink offset requests failing if there are still > active members in the consumer group. > > I don't believe logging an error message is sufficient for handling > failures to reset-after-delete. IMO it's highly likely that users will > either shoot themselves in the foot by not reading the fine print and > realizing that the offset request may have failed, or will ask for better > visibility into the success or failure of the reset request than scanning > log files. I don't doubt that there are ways to address this, but I would > prefer to leave them to a separate KIP since the required design work is > non-trivial and I do not feel that the added burden is worth tying to this > KIP as a blocker. > > I was really hoping to avoid introducing a change to the developer-facing > APIs with this KIP, but after giving it some thought I think this may be > unavoidable. It's debatable whether validation of altered offsets is a good > enough use case on its own for this kind of API, but since there are also > connectors out there that manage offsets externally, we should probably add > a hook to allow those external offsets to be managed, which can then serve > double- or even-triple duty as a hook to validate custom offsets and to > notify users whether offset resets/alterations are supported at all (which > they may not be if, for example, offsets are coupled tightly with the data > written by a sink connector). I've updated the KIP with the > developer-facing API changes for this logic; let me know what you think. > > Cheers, > > Chris > > On Mon, Nov 14, 2022 at 10:16 AM Mickael Maison > wrote: > > > Hi Chris, > > > > Thanks for the update! > > > > It's relatively common to only want to reset offsets for a specific > > resource (for example with MirrorMaker for one or a group of topics). > > Could it be possible to add a way to do so? Either by providing a > > payload to DELETE or by setting the offset field to an empty object in > > the PATCH payload? > > > > Thanks, > > Mickael > > > > On Sat, Nov 12, 2022 at 3:33 PM Yash Mayya wrote: > > > > > > Hi Chris, > > > > > > Thanks for pointing out that the consumer group deletion step itself > will > > > fail in case of zombie sink tasks. Since we can't get any stronger > > > guarantees from consumers (unlike with transactional producers), I > think > > it > > > makes perfect sense to fail the offset reset attempt in such scenarios > > with > > > a relevant error message to the user. I was more concerned about > silently > > > failing but it looks like that won't be an issue. It's probably worth > > > calling out this difference between source / sink connectors explicitly > > in > > > the KIP, what do you think? > > > > > > > changing the field names for sink offsets > > > > from "topic", "partition", and "offset" to "Kafka > > > > topic", "Kafka partition", and "Kafka offset" respectively, to > > > > reduce the stuttering effect of having a "partition" field inside > > > > a "partition" field and the same with an "offset" field > > > > > > The KIP is still using the nested partition / offset fields by the way > - > > > has it not been updated because we're waiting for consensus on the > field > > > names? > > > > > > > The reset-after-delete feature, on the other > > > > hand, is actually pretty tricky to design; I've updated the > > > > rationale in the KIP for delaying it and clarified that it's not > > > > just a matter of implementation but also design work. > > > > > > I like the idea of writing an offset reset request to the config topic > > > which will be processed by the herder's config update listener - I'm > not > > > sure I fully follow the concerns with regard to handling failures? Why > > > can't we simply log an error saying that
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Chris, Thanks for all the updates, yes that seems good! Mickael On Thu, Nov 17, 2022 at 8:41 PM Chris Egerton wrote: > > Hi Mickael, > > Thanks for your thoughts! IMO it's most intuitive to use a null value in > the PATCH API to signify that an offset should be reset, since it aligns > nicely with the API we provide to source connectors, where null offsets are > translated under the hood to tombstone records in the internal offsets > topic. Does that seem reasonable to you? > > Cheers, > > Chris > > On Thu, Nov 17, 2022 at 2:35 PM Chris Egerton wrote: > > > Hi Yash, > > > > I've updated the KIP with the correct "kafka_topic", "kafka_partition", > > and "kafka_offset" keys in the JSON examples (settled on those instead of > > prefixing with "Kafka " for better interactions with tooling like JQ). I've > > also added a note about sink offset requests failing if there are still > > active members in the consumer group. > > > > I don't believe logging an error message is sufficient for handling > > failures to reset-after-delete. IMO it's highly likely that users will > > either shoot themselves in the foot by not reading the fine print and > > realizing that the offset request may have failed, or will ask for better > > visibility into the success or failure of the reset request than scanning > > log files. I don't doubt that there are ways to address this, but I would > > prefer to leave them to a separate KIP since the required design work is > > non-trivial and I do not feel that the added burden is worth tying to this > > KIP as a blocker. > > > > I was really hoping to avoid introducing a change to the developer-facing > > APIs with this KIP, but after giving it some thought I think this may be > > unavoidable. It's debatable whether validation of altered offsets is a good > > enough use case on its own for this kind of API, but since there are also > > connectors out there that manage offsets externally, we should probably add > > a hook to allow those external offsets to be managed, which can then serve > > double- or even-triple duty as a hook to validate custom offsets and to > > notify users whether offset resets/alterations are supported at all (which > > they may not be if, for example, offsets are coupled tightly with the data > > written by a sink connector). I've updated the KIP with the > > developer-facing API changes for this logic; let me know what you think. > > > > Cheers, > > > > Chris > > > > On Mon, Nov 14, 2022 at 10:16 AM Mickael Maison > > wrote: > > > >> Hi Chris, > >> > >> Thanks for the update! > >> > >> It's relatively common to only want to reset offsets for a specific > >> resource (for example with MirrorMaker for one or a group of topics). > >> Could it be possible to add a way to do so? Either by providing a > >> payload to DELETE or by setting the offset field to an empty object in > >> the PATCH payload? > >> > >> Thanks, > >> Mickael > >> > >> On Sat, Nov 12, 2022 at 3:33 PM Yash Mayya wrote: > >> > > >> > Hi Chris, > >> > > >> > Thanks for pointing out that the consumer group deletion step itself > >> will > >> > fail in case of zombie sink tasks. Since we can't get any stronger > >> > guarantees from consumers (unlike with transactional producers), I > >> think it > >> > makes perfect sense to fail the offset reset attempt in such scenarios > >> with > >> > a relevant error message to the user. I was more concerned about > >> silently > >> > failing but it looks like that won't be an issue. It's probably worth > >> > calling out this difference between source / sink connectors explicitly > >> in > >> > the KIP, what do you think? > >> > > >> > > changing the field names for sink offsets > >> > > from "topic", "partition", and "offset" to "Kafka > >> > > topic", "Kafka partition", and "Kafka offset" respectively, to > >> > > reduce the stuttering effect of having a "partition" field inside > >> > > a "partition" field and the same with an "offset" field > >> > > >> > The KIP is still using the nested partition / offset fields by the way - > >> > has it not been updated because we're waiting for consensus on the field > >> > names? > >> > > >> > > The reset-after-delete feature, on the other > >> > > hand, is actually pretty tricky to design; I've updated the > >> > > rationale in the KIP for delaying it and clarified that it's not > >> > > just a matter of implementation but also design work. > >> > > >> > I like the idea of writing an offset reset request to the config topic > >> > which will be processed by the herder's config update listener - I'm not > >> > sure I fully follow the concerns with regard to handling failures? Why > >> > can't we simply log an error saying that the offset reset for the > >> deleted > >> > connector "xyz" failed due to reason "abc"? As long as it's documented > >> that > >> > connector deletion and offset resets are asynchronous and a success > >> > response only means that the request was initiated successfully (which > >> is
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Mickael, Thanks for your thoughts! IMO it's most intuitive to use a null value in the PATCH API to signify that an offset should be reset, since it aligns nicely with the API we provide to source connectors, where null offsets are translated under the hood to tombstone records in the internal offsets topic. Does that seem reasonable to you? Cheers, Chris On Thu, Nov 17, 2022 at 2:35 PM Chris Egerton wrote: > Hi Yash, > > I've updated the KIP with the correct "kafka_topic", "kafka_partition", > and "kafka_offset" keys in the JSON examples (settled on those instead of > prefixing with "Kafka " for better interactions with tooling like JQ). I've > also added a note about sink offset requests failing if there are still > active members in the consumer group. > > I don't believe logging an error message is sufficient for handling > failures to reset-after-delete. IMO it's highly likely that users will > either shoot themselves in the foot by not reading the fine print and > realizing that the offset request may have failed, or will ask for better > visibility into the success or failure of the reset request than scanning > log files. I don't doubt that there are ways to address this, but I would > prefer to leave them to a separate KIP since the required design work is > non-trivial and I do not feel that the added burden is worth tying to this > KIP as a blocker. > > I was really hoping to avoid introducing a change to the developer-facing > APIs with this KIP, but after giving it some thought I think this may be > unavoidable. It's debatable whether validation of altered offsets is a good > enough use case on its own for this kind of API, but since there are also > connectors out there that manage offsets externally, we should probably add > a hook to allow those external offsets to be managed, which can then serve > double- or even-triple duty as a hook to validate custom offsets and to > notify users whether offset resets/alterations are supported at all (which > they may not be if, for example, offsets are coupled tightly with the data > written by a sink connector). I've updated the KIP with the > developer-facing API changes for this logic; let me know what you think. > > Cheers, > > Chris > > On Mon, Nov 14, 2022 at 10:16 AM Mickael Maison > wrote: > >> Hi Chris, >> >> Thanks for the update! >> >> It's relatively common to only want to reset offsets for a specific >> resource (for example with MirrorMaker for one or a group of topics). >> Could it be possible to add a way to do so? Either by providing a >> payload to DELETE or by setting the offset field to an empty object in >> the PATCH payload? >> >> Thanks, >> Mickael >> >> On Sat, Nov 12, 2022 at 3:33 PM Yash Mayya wrote: >> > >> > Hi Chris, >> > >> > Thanks for pointing out that the consumer group deletion step itself >> will >> > fail in case of zombie sink tasks. Since we can't get any stronger >> > guarantees from consumers (unlike with transactional producers), I >> think it >> > makes perfect sense to fail the offset reset attempt in such scenarios >> with >> > a relevant error message to the user. I was more concerned about >> silently >> > failing but it looks like that won't be an issue. It's probably worth >> > calling out this difference between source / sink connectors explicitly >> in >> > the KIP, what do you think? >> > >> > > changing the field names for sink offsets >> > > from "topic", "partition", and "offset" to "Kafka >> > > topic", "Kafka partition", and "Kafka offset" respectively, to >> > > reduce the stuttering effect of having a "partition" field inside >> > > a "partition" field and the same with an "offset" field >> > >> > The KIP is still using the nested partition / offset fields by the way - >> > has it not been updated because we're waiting for consensus on the field >> > names? >> > >> > > The reset-after-delete feature, on the other >> > > hand, is actually pretty tricky to design; I've updated the >> > > rationale in the KIP for delaying it and clarified that it's not >> > > just a matter of implementation but also design work. >> > >> > I like the idea of writing an offset reset request to the config topic >> > which will be processed by the herder's config update listener - I'm not >> > sure I fully follow the concerns with regard to handling failures? Why >> > can't we simply log an error saying that the offset reset for the >> deleted >> > connector "xyz" failed due to reason "abc"? As long as it's documented >> that >> > connector deletion and offset resets are asynchronous and a success >> > response only means that the request was initiated successfully (which >> is >> > the case even today with normal connector deletion), we should be fine >> > right? >> > >> > Thanks for adding the new PATCH endpoint to the KIP, I think it's a lot >> > more useful for this use case than a PUT endpoint would be! One thing >> > that I was thinking about with the new PATCH endpoint is that while we >> can >> >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Yash, I've updated the KIP with the correct "kafka_topic", "kafka_partition", and "kafka_offset" keys in the JSON examples (settled on those instead of prefixing with "Kafka " for better interactions with tooling like JQ). I've also added a note about sink offset requests failing if there are still active members in the consumer group. I don't believe logging an error message is sufficient for handling failures to reset-after-delete. IMO it's highly likely that users will either shoot themselves in the foot by not reading the fine print and realizing that the offset request may have failed, or will ask for better visibility into the success or failure of the reset request than scanning log files. I don't doubt that there are ways to address this, but I would prefer to leave them to a separate KIP since the required design work is non-trivial and I do not feel that the added burden is worth tying to this KIP as a blocker. I was really hoping to avoid introducing a change to the developer-facing APIs with this KIP, but after giving it some thought I think this may be unavoidable. It's debatable whether validation of altered offsets is a good enough use case on its own for this kind of API, but since there are also connectors out there that manage offsets externally, we should probably add a hook to allow those external offsets to be managed, which can then serve double- or even-triple duty as a hook to validate custom offsets and to notify users whether offset resets/alterations are supported at all (which they may not be if, for example, offsets are coupled tightly with the data written by a sink connector). I've updated the KIP with the developer-facing API changes for this logic; let me know what you think. Cheers, Chris On Mon, Nov 14, 2022 at 10:16 AM Mickael Maison wrote: > Hi Chris, > > Thanks for the update! > > It's relatively common to only want to reset offsets for a specific > resource (for example with MirrorMaker for one or a group of topics). > Could it be possible to add a way to do so? Either by providing a > payload to DELETE or by setting the offset field to an empty object in > the PATCH payload? > > Thanks, > Mickael > > On Sat, Nov 12, 2022 at 3:33 PM Yash Mayya wrote: > > > > Hi Chris, > > > > Thanks for pointing out that the consumer group deletion step itself will > > fail in case of zombie sink tasks. Since we can't get any stronger > > guarantees from consumers (unlike with transactional producers), I think > it > > makes perfect sense to fail the offset reset attempt in such scenarios > with > > a relevant error message to the user. I was more concerned about silently > > failing but it looks like that won't be an issue. It's probably worth > > calling out this difference between source / sink connectors explicitly > in > > the KIP, what do you think? > > > > > changing the field names for sink offsets > > > from "topic", "partition", and "offset" to "Kafka > > > topic", "Kafka partition", and "Kafka offset" respectively, to > > > reduce the stuttering effect of having a "partition" field inside > > > a "partition" field and the same with an "offset" field > > > > The KIP is still using the nested partition / offset fields by the way - > > has it not been updated because we're waiting for consensus on the field > > names? > > > > > The reset-after-delete feature, on the other > > > hand, is actually pretty tricky to design; I've updated the > > > rationale in the KIP for delaying it and clarified that it's not > > > just a matter of implementation but also design work. > > > > I like the idea of writing an offset reset request to the config topic > > which will be processed by the herder's config update listener - I'm not > > sure I fully follow the concerns with regard to handling failures? Why > > can't we simply log an error saying that the offset reset for the deleted > > connector "xyz" failed due to reason "abc"? As long as it's documented > that > > connector deletion and offset resets are asynchronous and a success > > response only means that the request was initiated successfully (which is > > the case even today with normal connector deletion), we should be fine > > right? > > > > Thanks for adding the new PATCH endpoint to the KIP, I think it's a lot > > more useful for this use case than a PUT endpoint would be! One thing > > that I was thinking about with the new PATCH endpoint is that while we > can > > easily validate the request body format for sink connectors (since it's > the > > same across all connectors), we can't do the same for source connectors > as > > things stand today since each source connector implementation can define > > its own source partition and offset structures. Without any validation, > > writing a bad offset for a source connector via the PATCH endpoint could > > cause it to fail with hard to discern errors. I'm wondering if we could > add > > a new method to the `SourceConnector` class (which should be overridden > by > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
now,
> > > but I
> > > > > > may
> > > > > > > be
> > > > > > > > missing a case where a few extra records from a task that's
> > slow
> > > to
> > > > > > shut
> > > > > > > > down may cause serious issues--let me know.
> > > > > > > >
> > > > > > > > 6. I'm hesitant to propose deprecation of the PAUSED state
> > right
> > > now
> > > > > as
> > > > > > > it
> > > > > > > > does serve a few purposes. Leaving tasks idling-but-ready makes
> > > > > > resuming
> > > > > > > > them less disruptive across the cluster, since a rebalance
> > isn't
> > > > > > > necessary.
> > > > > > > > It also reduces latency to resume the connector, especially for
> > > ones
> > > > > > that
> > > > > > > > have to do a lot of state gathering on initialization to, e.g.,
> > > read
> > > > > > > > offsets from an external system.
> > > > > > > >
> > > > > > > > 7. There should be no risk of mixed tasks after a downgrade,
> > > thanks
> > > > > to
> > > > > > > the
> > > > > > > > empty set of task configs that gets published to the config
> > > topic.
> > > > > Both
> > > > > > > > upgraded and downgraded workers will render an empty set of
> > > tasks for
> > > > > > the
> > > > > > > > connector, and keep that set of empty tasks until the connector
> > > is
> > > > > > > resumed.
> > > > > > > > Does that address your concerns?
> > > > > > > >
> > > > > > > > You're also correct that the linked Jira ticket was wrong;
> > > thanks for
> > > > > > > > pointing that out! Yes, KAFKA-4107 is the intended ticket, and
> > > I've
> > > > > > > updated
> > > > > > > > the link in the KIP accordingly.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > [1] - https://issues.apache.org/jira/browse/KAFKA-4107
> > > > > > > >
> > > > > > > > On Sun, Oct 16, 2022 at 10:42 AM Yash Mayya <
> > > yash.ma...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > Thanks a lot for this KIP, I think something like this has
> > been
> > > > > long
> > > > > > > > > overdue for Kafka Connect :)
> > > > > > > > >
> > > > > > > > > Some thoughts and questions that I had -
> > > > > > > > >
> > > > > > > > > 1. I'm wondering if you could elaborate a little more on the
> > > use
> > > > > case
> > > > > > > for
> > > > > > > > > the `DELETE /connectors/{connector}/offsets` API. I think we
> > > can
> > > > > all
> > > > > > > > agree
> > > > > > > > > that a fine grained reset API that allows setting arbitrary
> > > offsets
> > > > > > for
> > > > > > > > > partitions would be quite useful (which you talk about in the
> > > > > Future
> > > > > > > work
> > > > > > > > > section). But for the `DELETE
> > /connectors/{connector}/offsets`
> > > API
> > > > > in
> > > > > > > its
> > > > > > > > > described form, it looks like it would only serve a seemingly
> > > niche
> > > > > > use
> > > > > > > > > case where users want to avoid renaming connectors - because
> > > this
> > > > > new
> > > > > > > way
> > > > > > > > > of resetting offsets actually has more steps (i.e. stop the
> > > > > > connector,
> > > > > > > > > reset offsets via the API, res
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
be stuck on a long-running
> > > > > > SinkTask::put
> > > > > > > that emits data even after the Connect framework has abandoned
> > them
> > > > > after
> > > > > > > exhausting their graceful shutdown timeout. Ultimately, I'd
> > prefer to
> > > > > err
> > > > > > > on the side of consistency and ease of implementation for now,
> > but I
> > > > > may
> > > > > > be
> > > > > > > missing a case where a few extra records from a task that's
> slow
> > to
> > > > > shut
> > > > > > > down may cause serious issues--let me know.
> > > > > > >
> > > > > > > 6. I'm hesitant to propose deprecation of the PAUSED state
> right
> > now
> > > > as
> > > > > > it
> > > > > > > does serve a few purposes. Leaving tasks idling-but-ready makes
> > > > > resuming
> > > > > > > them less disruptive across the cluster, since a rebalance
> isn't
> > > > > > necessary.
> > > > > > > It also reduces latency to resume the connector, especially for
> > ones
> > > > > that
> > > > > > > have to do a lot of state gathering on initialization to, e.g.,
> > read
> > > > > > > offsets from an external system.
> > > > > > >
> > > > > > > 7. There should be no risk of mixed tasks after a downgrade,
> > thanks
> > > > to
> > > > > > the
> > > > > > > empty set of task configs that gets published to the config
> > topic.
> > > > Both
> > > > > > > upgraded and downgraded workers will render an empty set of
> > tasks for
> > > > > the
> > > > > > > connector, and keep that set of empty tasks until the connector
> > is
> > > > > > resumed.
> > > > > > > Does that address your concerns?
> > > > > > >
> > > > > > > You're also correct that the linked Jira ticket was wrong;
> > thanks for
> > > > > > > pointing that out! Yes, KAFKA-4107 is the intended ticket, and
> > I've
> > > > > > updated
> > > > > > > the link in the KIP accordingly.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > [1] - https://issues.apache.org/jira/browse/KAFKA-4107
> > > > > > >
> > > > > > > On Sun, Oct 16, 2022 at 10:42 AM Yash Mayya <
> > yash.ma...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > Thanks a lot for this KIP, I think something like this has
> been
> > > > long
> > > > > > > > overdue for Kafka Connect :)
> > > > > > > >
> > > > > > > > Some thoughts and questions that I had -
> > > > > > > >
> > > > > > > > 1. I'm wondering if you could elaborate a little more on the
> > use
> > > > case
> > > > > > for
> > > > > > > > the `DELETE /connectors/{connector}/offsets` API. I think we
> > can
> > > > all
> > > > > > > agree
> > > > > > > > that a fine grained reset API that allows setting arbitrary
> > offsets
> > > > > for
> > > > > > > > partitions would be quite useful (which you talk about in the
> > > > Future
> > > > > > work
> > > > > > > > section). But for the `DELETE
> /connectors/{connector}/offsets`
> > API
> > > > in
> > > > > > its
> > > > > > > > described form, it looks like it would only serve a seemingly
> > niche
> > > > > use
> > > > > > > > case where users want to avoid renaming connectors - because
> > this
> > > > new
> > > > > > way
> > > > > > > > of resetting offsets actually has more steps (i.e. stop the
> > > > > connector,
> > > > > > > > reset offsets via the API, resume the connector) than sim
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
a ticket was wrong;
> thanks for
> > > > > > pointing that out! Yes, KAFKA-4107 is the intended ticket, and
> I've
> > > > > updated
> > > > > > the link in the KIP accordingly.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > [1] - https://issues.apache.org/jira/browse/KAFKA-4107
> > > > > >
> > > > > > On Sun, Oct 16, 2022 at 10:42 AM Yash Mayya <
> yash.ma...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > Thanks a lot for this KIP, I think something like this has been
> > > long
> > > > > > > overdue for Kafka Connect :)
> > > > > > >
> > > > > > > Some thoughts and questions that I had -
> > > > > > >
> > > > > > > 1. I'm wondering if you could elaborate a little more on the
> use
> > > case
> > > > > for
> > > > > > > the `DELETE /connectors/{connector}/offsets` API. I think we
> can
> > > all
> > > > > > agree
> > > > > > > that a fine grained reset API that allows setting arbitrary
> offsets
> > > > for
> > > > > > > partitions would be quite useful (which you talk about in the
> > > Future
> > > > > work
> > > > > > > section). But for the `DELETE /connectors/{connector}/offsets`
> API
> > > in
> > > > > its
> > > > > > > described form, it looks like it would only serve a seemingly
> niche
> > > > use
> > > > > > > case where users want to avoid renaming connectors - because
> this
> > > new
> > > > > way
> > > > > > > of resetting offsets actually has more steps (i.e. stop the
> > > > connector,
> > > > > > > reset offsets via the API, resume the connector) than simply
> > > deleting
> > > > > and
> > > > > > > re-creating the connector with a different name?
> > > > > > >
> > > > > > > 2. The KIP talks about taking care that the response formats
> > > > > (presumably
> > > > > > > only talking about the new GET API here) are symmetrical for
> both
> > > > > source
> > > > > > > and sink connectors - is the end goal to have users of Kafka
> > > Connect
> > > > > not
> > > > > > > even be aware that sink connectors use Kafka consumers under
> the
> > > hood
> > > > > > (i.e.
> > > > > > > have that as purely an implementation detail abstracted away
> from
> > > > > users)?
> > > > > > > While I understand the value of uniformity here, the response
> > > format
> > > > > for
> > > > > > > sink connectors currently looks a little odd with the
> "partition"
> > > > field
> > > > > > > having "topic" and "partition" as sub-fields, especially to
> users
> > > > > > familiar
> > > > > > > with Kafka semantics. Thoughts?
> > > > > > >
> > > > > > > 3. Another little nitpick on the response format - why do we
> need
> > > > > > "source"
> > > > > > > / "sink" as a field under "offsets"? Users can query the
> connector
> > > > type
> > > > > > via
> > > > > > > the existing `GET /connectors` API. If it's deemed important
> to let
> > > > > users
> > > > > > > know that the offsets they're seeing correspond to a source /
> sink
> > > > > > > connector, maybe we could have a top level field "type" in the
> > > > response
> > > > > > for
> > > > > > > the `GET /connectors/{connector}/offsets` API similar to the
> `GET
> > > > > > > /connectors` API?
> > > > > > >
> > > > > > > 4. For the `DELETE /connectors/{connector}/offsets` API, the
> KIP
> > > > > mentions
> > > > > > > that requests will be rejected if a rebalance is pending -
> > > presuma
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
s generated, rather than
> > > the
> > > > > > delete offsets endpoint? This would also give the new `STOPPED`
> > > state a
> > > > > > higher confidence of sorts, with any zombie tasks being fenced off
> > > from
> > > > > > continuing to produce data.
> > > > > >
> > > > > > 6. Thanks for outlining the issues with the current state of the
> > > > `PAUSED`
> > > > > > state - I think a lot of users expect it to behave like the
> > `STOPPED`
> > > > > state
> > > > > > you outline in the KIP and are (unpleasantly) surprised when it
> > > > doesn't.
> > > > > > However, this does beg the question of what the usefulness of
> > having
> > > > two
> > > > > > separate `PAUSED` and `STOPPED` states is? Do we want to continue
> > > > > > supporting both these states in the future, or do you see the
> > > `STOPPED`
> > > > > > state eventually causing the existing `PAUSED` state to be
> > > deprecated?
> > > > > >
> > > > > > 7. I think the idea outlined in the KIP for handling a new state
> > > during
> > > > > > cluster downgrades / rolling upgrades is quite clever, but do you
> > > think
> > > > > > there could be any issues with having a mix of "paused" and
> > "stopped"
> > > > > tasks
> > > > > > for the same connector across workers in a cluster? At the very
> > > least,
> > > > I
> > > > > > think it would be fairly confusing to most users. I'm wondering if
> > > this
> > > > > can
> > > > > > be avoided by stating clearly in the KIP that the new `PUT
> > > > > > /connectors/{connector}/stop`
> > > > > > can only be used on a cluster that is fully upgraded to an AK
> > version
> > > > > newer
> > > > > > than the one which ends up containing changes from this KIP and
> > that
> > > > if a
> > > > > > cluster needs to be downgraded to an older version, the user should
> > > > > ensure
> > > > > > that none of the connectors on the cluster are in a stopped state?
> > > With
> > > > > the
> > > > > > existing implementation, it looks like an unknown/invalid target
> > > state
> > > > > > record is basically just discarded (with an error message logged),
> > so
> > > > it
> > > > > > doesn't seem to be a disastrous failure scenario that can bring
> > down
> > > a
> > > > > > worker.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Yash
> > > > > >
> > > > > > On Fri, Oct 14, 2022 at 8:35 PM Chris Egerton
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Ashwin,
> > > > > > >
> > > > > > > Thanks for your thoughts. Regarding your questions:
> > > > > > >
> > > > > > > 1. The response would show the offsets that are visible to the
> > > source
> > > > > > > connector, so it would combine the contents of the two topics,
> > > giving
> > > > > > > priority to offsets present in the connector-specific topic. I'm
> > > > > > imagining
> > > > > > > a follow-up question that some people may have in response to
> > that
> > > is
> > > > > > > whether we'd want to provide insight into the contents of a
> > single
> > > > > topic
> > > > > > at
> > > > > > > a time. It may be useful to be able to see this information in
> > > order
> > > > to
> > > > > > > debug connector issues or verify that it's safe to stop using a
> > > > > > > connector-specific offsets topic (either explicitly, or
> > implicitly
> > > > via
> > > > > > > cluster downgrade). What do you think about adding a URL query
> > > > > parameter
> > > > > > > that allows users to dictate which view of the connector's
> > offsets
> > > > they
> > > > > > are
> > > > > > > given in the
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
or}/offsets` API similar to the `GET
> > > > > /connectors` API?
> > > > >
> > > > > 4. For the `DELETE /connectors/{connector}/offsets` API, the KIP
> > > mentions
> > > > > that requests will be rejected if a rebalance is pending -
> presumably
> > > > this
> > > > > is to avoid forwarding requests to a leader which may no longer be
> > the
> > > > > leader after the pending rebalance? In this case, the API will
> > return a
> > > > > `409 Conflict` response similar to some of the existing APIs,
> right?
> > > > >
> > > > > 5. Regarding fencing out previously running tasks for a connector,
> do
> > > you
> > > > > think it would make more sense semantically to have this
> implemented
> > in
> > > > the
> > > > > stop endpoint where an empty set of tasks is generated, rather than
> > the
> > > > > delete offsets endpoint? This would also give the new `STOPPED`
> > state a
> > > > > higher confidence of sorts, with any zombie tasks being fenced off
> > from
> > > > > continuing to produce data.
> > > > >
> > > > > 6. Thanks for outlining the issues with the current state of the
> > > `PAUSED`
> > > > > state - I think a lot of users expect it to behave like the
> `STOPPED`
> > > > state
> > > > > you outline in the KIP and are (unpleasantly) surprised when it
> > > doesn't.
> > > > > However, this does beg the question of what the usefulness of
> having
> > > two
> > > > > separate `PAUSED` and `STOPPED` states is? Do we want to continue
> > > > > supporting both these states in the future, or do you see the
> > `STOPPED`
> > > > > state eventually causing the existing `PAUSED` state to be
> > deprecated?
> > > > >
> > > > > 7. I think the idea outlined in the KIP for handling a new state
> > during
> > > > > cluster downgrades / rolling upgrades is quite clever, but do you
> > think
> > > > > there could be any issues with having a mix of "paused" and
> "stopped"
> > > > tasks
> > > > > for the same connector across workers in a cluster? At the very
> > least,
> > > I
> > > > > think it would be fairly confusing to most users. I'm wondering if
> > this
> > > > can
> > > > > be avoided by stating clearly in the KIP that the new `PUT
> > > > > /connectors/{connector}/stop`
> > > > > can only be used on a cluster that is fully upgraded to an AK
> version
> > > > newer
> > > > > than the one which ends up containing changes from this KIP and
> that
> > > if a
> > > > > cluster needs to be downgraded to an older version, the user should
> > > > ensure
> > > > > that none of the connectors on the cluster are in a stopped state?
> > With
> > > > the
> > > > > existing implementation, it looks like an unknown/invalid target
> > state
> > > > > record is basically just discarded (with an error message logged),
> so
> > > it
> > > > > doesn't seem to be a disastrous failure scenario that can bring
> down
> > a
> > > > > worker.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Yash
> > > > >
> > > > > On Fri, Oct 14, 2022 at 8:35 PM Chris Egerton
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Ashwin,
> > > > > >
> > > > > > Thanks for your thoughts. Regarding your questions:
> > > > > >
> > > > > > 1. The response would show the offsets that are visible to the
> > source
> > > > > > connector, so it would combine the contents of the two topics,
> > giving
> > > > > > priority to offsets present in the connector-specific topic. I'm
> > > > > imagining
> > > > > > a follow-up question that some people may have in response to
> that
> > is
> > > > > > whether we'd want to provide insight into the contents of a
> single
> > > > topic
> > > > > at
> > > > > > a time. It may be useful to be able to see this information in
> > order
> > > to
> > > > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
xisting `GET /connectors` API. If it's deemed important to let
> > users
> > > > know that the offsets they're seeing correspond to a source / sink
> > > > connector, maybe we could have a top level field "type" in the
> response
> > > for
> > > > the `GET /connectors/{connector}/offsets` API similar to the `GET
> > > > /connectors` API?
> > > >
> > > > 4. For the `DELETE /connectors/{connector}/offsets` API, the KIP
> > mentions
> > > > that requests will be rejected if a rebalance is pending - presumably
> > > this
> > > > is to avoid forwarding requests to a leader which may no longer be
> the
> > > > leader after the pending rebalance? In this case, the API will
> return a
> > > > `409 Conflict` response similar to some of the existing APIs, right?
> > > >
> > > > 5. Regarding fencing out previously running tasks for a connector, do
> > you
> > > > think it would make more sense semantically to have this implemented
> in
> > > the
> > > > stop endpoint where an empty set of tasks is generated, rather than
> the
> > > > delete offsets endpoint? This would also give the new `STOPPED`
> state a
> > > > higher confidence of sorts, with any zombie tasks being fenced off
> from
> > > > continuing to produce data.
> > > >
> > > > 6. Thanks for outlining the issues with the current state of the
> > `PAUSED`
> > > > state - I think a lot of users expect it to behave like the `STOPPED`
> > > state
> > > > you outline in the KIP and are (unpleasantly) surprised when it
> > doesn't.
> > > > However, this does beg the question of what the usefulness of having
> > two
> > > > separate `PAUSED` and `STOPPED` states is? Do we want to continue
> > > > supporting both these states in the future, or do you see the
> `STOPPED`
> > > > state eventually causing the existing `PAUSED` state to be
> deprecated?
> > > >
> > > > 7. I think the idea outlined in the KIP for handling a new state
> during
> > > > cluster downgrades / rolling upgrades is quite clever, but do you
> think
> > > > there could be any issues with having a mix of "paused" and "stopped"
> > > tasks
> > > > for the same connector across workers in a cluster? At the very
> least,
> > I
> > > > think it would be fairly confusing to most users. I'm wondering if
> this
> > > can
> > > > be avoided by stating clearly in the KIP that the new `PUT
> > > > /connectors/{connector}/stop`
> > > > can only be used on a cluster that is fully upgraded to an AK version
> > > newer
> > > > than the one which ends up containing changes from this KIP and that
> > if a
> > > > cluster needs to be downgraded to an older version, the user should
> > > ensure
> > > > that none of the connectors on the cluster are in a stopped state?
> With
> > > the
> > > > existing implementation, it looks like an unknown/invalid target
> state
> > > > record is basically just discarded (with an error message logged), so
> > it
> > > > doesn't seem to be a disastrous failure scenario that can bring down
> a
> > > > worker.
> > > >
> > > >
> > > > Thanks,
> > > > Yash
> > > >
> > > > On Fri, Oct 14, 2022 at 8:35 PM Chris Egerton
> > >
> > > > wrote:
> > > >
> > > > > Hi Ashwin,
> > > > >
> > > > > Thanks for your thoughts. Regarding your questions:
> > > > >
> > > > > 1. The response would show the offsets that are visible to the
> source
> > > > > connector, so it would combine the contents of the two topics,
> giving
> > > > > priority to offsets present in the connector-specific topic. I'm
> > > > imagining
> > > > > a follow-up question that some people may have in response to that
> is
> > > > > whether we'd want to provide insight into the contents of a single
> > > topic
> > > > at
> > > > > a time. It may be useful to be able to see this information in
> order
> > to
> > > > > debug connector issues or verify that it's safe to stop using a
> > > > > connector-specific offsets topic (either explicitly, or implicitly
> > via
> > > > > cluster downgrade). What do you t
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
ther we'd want to provide insight into the contents of a single
> > topic
> > > at
> > > > a time. It may be useful to be able to see this information in order
> to
> > > > debug connector issues or verify that it's safe to stop using a
> > > > connector-specific offsets topic (either explicitly, or implicitly
> via
> > > > cluster downgrade). What do you think about adding a URL query
> > parameter
> > > > that allows users to dictate which view of the connector's offsets
> they
> > > are
> > > > given in the REST response, with options for the worker's global
> topic,
> > > the
> > > > connector-specific topic, and the combined view of them that the
> > > connector
> > > > and its tasks see (which would be the default)? This may be too much
> > for
> > > V1
> > > > but it feels like it's at least worth exploring a bit.
> > > >
> > > > 2. There is no option for this at the moment. Reset semantics are
> > > extremely
> > > > coarse-grained; for source connectors, we delete all source offsets,
> > and
> > > > for sink connectors, we delete the entire consumer group. I'm hoping
> > this
> > > > will be enough for V1 and that, if there's sufficient demand for it,
> we
> > > can
> > > > introduce a richer API for resetting or even modifying connector
> > offsets
> > > in
> > > > a follow-up KIP.
> > > >
> > > > 3. Good eye :) I think it's fine to keep the existing behavior for
> the
> > > > PAUSED state with the Connector instance, since the primary purpose
> of
> > > the
> > > > Connector is to generate task configs and monitor the external system
> > for
> > > > changes. If there's no chance for tasks to be running anyways, I
> don't
> > > see
> > > > much value in allowing paused connectors to generate new task
> configs,
> > > > especially since each time that happens a rebalance is triggered and
> > > > there's a non-zero cost to that. What do you think?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Fri, Oct 14, 2022 at 12:59 AM Ashwin >
> > > > wrote:
> > > >
> > > > > Thanks for KIP Chris - I think this is a useful feature.
> > > > >
> > > > > Can you please elaborate on the following in the KIP -
> > > > >
> > > > > 1. How would the response of GET /connectors/{connector}/offsets
> look
> > > > like
> > > > > if the worker has both global and connector specific offsets topic
> ?
> > > > >
> > > > > 2. How can we pass the reset options like shift-by , to-date-time
> > etc.
> > > > > using a REST API like DELETE /connectors/{connector}/offsets ?
> > > > >
> > > > > 3. Today PAUSE operation on a connector invokes its stop method -
> > will
> > > > > there be a change here to reduce confusion with the new proposed
> > > STOPPED
> > > > > state ?
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > > > On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I noticed a fairly large gap in the first version of this KIP
> that
> > I
> > > > > > published last Friday, which has to do with accommodating
> > connectors
> > > > > > that target different Kafka clusters than the one that the Kafka
> > > > Connect
> > > > > > cluster uses for its internal topics and source connectors with
> > > > dedicated
> > > > > > offsets topics. I've since updated the KIP to address this gap,
> > which
> > > > has
> > > > > > substantially altered the design. Wanted to give a heads-up to
> > anyone
> > > > > > that's already started reviewing.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton
> > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to begin discussion on a KIP to add offsets support to
> > the
> > > > > Kafka
> > > > > > > Connect REST API:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
nce the primary purpose of
> > the
> > > Connector is to generate task configs and monitor the external system
> for
> > > changes. If there's no chance for tasks to be running anyways, I don't
> > see
> > > much value in allowing paused connectors to generate new task configs,
> > > especially since each time that happens a rebalance is triggered and
> > > there's a non-zero cost to that. What do you think?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Fri, Oct 14, 2022 at 12:59 AM Ashwin
> > > wrote:
> > >
> > > > Thanks for KIP Chris - I think this is a useful feature.
> > > >
> > > > Can you please elaborate on the following in the KIP -
> > > >
> > > > 1. How would the response of GET /connectors/{connector}/offsets look
> > > like
> > > > if the worker has both global and connector specific offsets topic ?
> > > >
> > > > 2. How can we pass the reset options like shift-by , to-date-time
> etc.
> > > > using a REST API like DELETE /connectors/{connector}/offsets ?
> > > >
> > > > 3. Today PAUSE operation on a connector invokes its stop method -
> will
> > > > there be a change here to reduce confusion with the new proposed
> > STOPPED
> > > > state ?
> > > >
> > > > Thanks,
> > > > Ashwin
> > > >
> > > > On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
> > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I noticed a fairly large gap in the first version of this KIP that
> I
> > > > > published last Friday, which has to do with accommodating
> connectors
> > > > > that target different Kafka clusters than the one that the Kafka
> > > Connect
> > > > > cluster uses for its internal topics and source connectors with
> > > dedicated
> > > > > offsets topics. I've since updated the KIP to address this gap,
> which
> > > has
> > > > > substantially altered the design. Wanted to give a heads-up to
> anyone
> > > > > that's already started reviewing.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to begin discussion on a KIP to add offsets support to
> the
> > > > Kafka
> > > > > > Connect REST API:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
g both these states in the future, or do you see the `STOPPED`
> state eventually causing the existing `PAUSED` state to be deprecated?
>
> 7. I think the idea outlined in the KIP for handling a new state during
> cluster downgrades / rolling upgrades is quite clever, but do you think
> there could be any issues with having a mix of "paused" and "stopped" tasks
> for the same connector across workers in a cluster? At the very least, I
> think it would be fairly confusing to most users. I'm wondering if this can
> be avoided by stating clearly in the KIP that the new `PUT
> /connectors/{connector}/stop`
> can only be used on a cluster that is fully upgraded to an AK version newer
> than the one which ends up containing changes from this KIP and that if a
> cluster needs to be downgraded to an older version, the user should ensure
> that none of the connectors on the cluster are in a stopped state? With the
> existing implementation, it looks like an unknown/invalid target state
> record is basically just discarded (with an error message logged), so it
> doesn't seem to be a disastrous failure scenario that can bring down a
> worker.
>
>
> Thanks,
> Yash
>
> On Fri, Oct 14, 2022 at 8:35 PM Chris Egerton
> wrote:
>
> > Hi Ashwin,
> >
> > Thanks for your thoughts. Regarding your questions:
> >
> > 1. The response would show the offsets that are visible to the source
> > connector, so it would combine the contents of the two topics, giving
> > priority to offsets present in the connector-specific topic. I'm
> imagining
> > a follow-up question that some people may have in response to that is
> > whether we'd want to provide insight into the contents of a single topic
> at
> > a time. It may be useful to be able to see this information in order to
> > debug connector issues or verify that it's safe to stop using a
> > connector-specific offsets topic (either explicitly, or implicitly via
> > cluster downgrade). What do you think about adding a URL query parameter
> > that allows users to dictate which view of the connector's offsets they
> are
> > given in the REST response, with options for the worker's global topic,
> the
> > connector-specific topic, and the combined view of them that the
> connector
> > and its tasks see (which would be the default)? This may be too much for
> V1
> > but it feels like it's at least worth exploring a bit.
> >
> > 2. There is no option for this at the moment. Reset semantics are
> extremely
> > coarse-grained; for source connectors, we delete all source offsets, and
> > for sink connectors, we delete the entire consumer group. I'm hoping this
> > will be enough for V1 and that, if there's sufficient demand for it, we
> can
> > introduce a richer API for resetting or even modifying connector offsets
> in
> > a follow-up KIP.
> >
> > 3. Good eye :) I think it's fine to keep the existing behavior for the
> > PAUSED state with the Connector instance, since the primary purpose of
> the
> > Connector is to generate task configs and monitor the external system for
> > changes. If there's no chance for tasks to be running anyways, I don't
> see
> > much value in allowing paused connectors to generate new task configs,
> > especially since each time that happens a rebalance is triggered and
> > there's a non-zero cost to that. What do you think?
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, Oct 14, 2022 at 12:59 AM Ashwin
> > wrote:
> >
> > > Thanks for KIP Chris - I think this is a useful feature.
> > >
> > > Can you please elaborate on the following in the KIP -
> > >
> > > 1. How would the response of GET /connectors/{connector}/offsets look
> > like
> > > if the worker has both global and connector specific offsets topic ?
> > >
> > > 2. How can we pass the reset options like shift-by , to-date-time etc.
> > > using a REST API like DELETE /connectors/{connector}/offsets ?
> > >
> > > 3. Today PAUSE operation on a connector invokes its stop method - will
> > > there be a change here to reduce confusion with the new proposed
> STOPPED
> > > state ?
> > >
> > > Thanks,
> > > Ashwin
> > >
> > > On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I noticed a fairly large gap in the first version of this KIP that I
> > > > published last Friday, which has to do with accommodating connectors
> > > > that target different Kafka clusters than the one that the Kafka
> > Connect
> > > > cluster uses for its internal topics and source connectors with
> > dedicated
> > > > offsets topics. I've since updated the KIP to address this gap, which
> > has
> > > > substantially altered the design. Wanted to give a heads-up to anyone
> > > > that's already started reviewing.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to begin discussion on a KIP to add offsets support to the
> > > Kafka
> > > > > Connect REST API:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
I realised that I mistook the KPI and didn't realize this wasn't part of the REST API so you can ignore what I previously wrote. On Mon, 17 Oct 2022, 12:18 Matthew Benedict de Detrich, < matthew.dedetr...@aiven.io> wrote: > So my only 2 cents on this KIP is mainly from a REST perspective, i.e. in > the KIP you mention that you need to add a new field "state.v2": “STOPPED” > due to maintaining backwards compatibility with older brokers. My main > qualm here is that putting versioning into JSON field names isn’t typically > the best from a REST design perspective. I would propose using something > like “current_state": “STOPPED” instead or anything other field name that > seems appropriate. > > There may be other alternatives as well but in summary there appears to be > a tradeoff between making sure the new design is clean from a REST > standpoint (which is where I am personally leaning) versus making the > transition easier/more understandable for older brokers. > > Regards, > > -- > > Matthew de Detrich > > *Aiven Deutschland GmbH* > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > *m:* +491603708037 > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > On 13. Oct 2022, 22:52 +0200, Chris Egerton , > wrote: > > Hi all, > > I noticed a fairly large gap in the first version of this KIP that I > published last Friday, which has to do with accommodating connectors > that target different Kafka clusters than the one that the Kafka Connect > cluster uses for its internal topics and source connectors with dedicated > offsets topics. I've since updated the KIP to address this gap, which has > substantially altered the design. Wanted to give a heads-up to anyone > that's already started reviewing. > > Cheers, > > Chris > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > Hi all, > > I'd like to begin discussion on a KIP to add offsets support to the Kafka > Connect REST API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > Cheers, > > Chris > >
Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
rtition) > > for sink connectors. What are your thoughts? If this strategy makes > sense I > > can add it to the future work section. > > > > 2. The new STOPPED state should "just work" with the rebalancing > algorithm, > > since it'll be implemented under the hood by publishing an empty set of > > task configs to the config topic for the connector. That should be enough > > to trigger a rebalance and to cause no tasks for the connector to be > > allocated across the cluster during that rebalance, regardless of the > > protocol (eager or incremental) that's in use. > > > > RE your implementation questions: > > > > 1. It's mostly a matter of convenience; we can issue a single admin > request > > to delete the group rather than having to identify every topic partition > > the consumer group has committed offsets for and then issue a follow-up > > request to delete the offsets for that group. I made note of this detail > in > > the KIP to make sure that we were comfortable with completely removing > the > > consumer group instead of wiping its offsets, since it seems possible > that > > some users may find that behavior unexpected. > > > > 2. The idea is to only perform zombie fencing when we know that it is > > necessary (which is a principle borrowed from KIP-618), so in this case, > > we'll only do it in response to an offsets reset request, and not when a > > connector is stopped. After being stopped, it's possible that the > connector > > gets deleted, in which case, a proactive round of fencing would have > served > > no benefit. It's also worth noting that publishing an empty set of task > > configs is not the same as tombstoning existing task configs; putting a > > connector into the STOPPED state should require no tombstones to be > emitted > > to the config topic. > > > > Cheers, > > > > Chris > > > > On Thu, Oct 13, 2022 at 6:26 PM Greg Harris > > > wrote: > > > > > Hey Chris, > > > > > > Thanks for the KIP! > > > > > > I think this is an important feature for both development and > operations > > > use-cases, and it's an obvious gap in the REST feature set. > > > I also appreciate the incremental nature of the KIP and the future > > > extensions that will now be possible. > > > > > > I had a couple of questions about the design and it's extensibility: > > > > > > 1. How do you imagine the API will behave with connectors that have > > > extremely large numbers of partitions (thousands or more) and/or source > > > connectors with large amounts of data per partition? > > > > > > 2. Does the new STOPPED state need any special integration with the > > > rebalance subsystem, or can the rebalance algorithms remain ignorant of > > the > > > target state of connectors? > > > > > > And about the implementation: > > > > > > 1. For my own edification, what is the difference between deleting a > > > consumer group and deleting all known offsets for that group? Does > > deleting > > > the group offer better/easier atomicity? > > > > > > 2. For EOS sources, will stopping the connector and tombstoning the > task > > > configs perform a fence-out, or will that fence-out only occur when > > > performing the offsets DELETE operation? > > > > > > Thanks! > > > Greg > > > > > > On 2022/10/13 20:52:26 Chris Egerton wrote: > > > > Hi all, > > > > > > > > I noticed a fairly large gap in the first version of this KIP that I > > > > published last Friday, which has to do with accommodating connectors > > > > that target different Kafka clusters than the one that the Kafka > > Connect > > > > cluster uses for its internal topics and source connectors with > > dedicated > > > > offsets topics. I've since updated the KIP to address this gap, which > > has > > > > substantially altered the design. Wanted to give a heads-up to anyone > > > > that's already started reviewing. > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to begin discussion on a KIP to add offsets support to the > > > Kafka > > > > > Connect REST API: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > > > Cheers, > > > > > > > > > > Chris > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
So my only 2 cents on this KIP is mainly from a REST perspective, i.e. in the KIP you mention that you need to add a new field "state.v2": “STOPPED” due to maintaining backwards compatibility with older brokers. My main qualm here is that putting versioning into JSON field names isn’t typically the best from a REST design perspective. I would propose using something like “current_state": “STOPPED” instead or anything other field name that seems appropriate. There may be other alternatives as well but in summary there appears to be a tradeoff between making sure the new design is clean from a REST standpoint (which is where I am personally leaning) versus making the transition easier/more understandable for older brokers. Regards, -- Matthew de Detrich Aiven Deutschland GmbH Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen m: +491603708037 w: aiven.io e: matthew.dedetr...@aiven.io On 13. Oct 2022, 22:52 +0200, Chris Egerton , wrote: > Hi all, > > I noticed a fairly large gap in the first version of this KIP that I > published last Friday, which has to do with accommodating connectors > that target different Kafka clusters than the one that the Kafka Connect > cluster uses for its internal topics and source connectors with dedicated > offsets topics. I've since updated the KIP to address this gap, which has > substantially altered the design. Wanted to give a heads-up to anyone > that's already started reviewing. > > Cheers, > > Chris > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > > Hi all, > > > > I'd like to begin discussion on a KIP to add offsets support to the Kafka > > Connect REST API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > Cheers, > > > > Chris > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
gt;> Thanks for your thoughts. Regarding your questions:
>>
>> 1. The response would show the offsets that are visible to the source
>> connector, so it would combine the contents of the two topics, giving
>> priority to offsets present in the connector-specific topic. I'm imagining
>> a follow-up question that some people may have in response to that is
>> whether we'd want to provide insight into the contents of a single topic
>> at
>> a time. It may be useful to be able to see this information in order to
>> debug connector issues or verify that it's safe to stop using a
>> connector-specific offsets topic (either explicitly, or implicitly via
>> cluster downgrade). What do you think about adding a URL query parameter
>> that allows users to dictate which view of the connector's offsets they
>> are
>> given in the REST response, with options for the worker's global topic,
>> the
>> connector-specific topic, and the combined view of them that the connector
>> and its tasks see (which would be the default)? This may be too much for
>> V1
>> but it feels like it's at least worth exploring a bit.
>>
>> 2. There is no option for this at the moment. Reset semantics are
>> extremely
>> coarse-grained; for source connectors, we delete all source offsets, and
>> for sink connectors, we delete the entire consumer group. I'm hoping this
>> will be enough for V1 and that, if there's sufficient demand for it, we
>> can
>> introduce a richer API for resetting or even modifying connector offsets
>> in
>> a follow-up KIP.
>>
>> 3. Good eye :) I think it's fine to keep the existing behavior for the
>> PAUSED state with the Connector instance, since the primary purpose of the
>> Connector is to generate task configs and monitor the external system for
>> changes. If there's no chance for tasks to be running anyways, I don't see
>> much value in allowing paused connectors to generate new task configs,
>> especially since each time that happens a rebalance is triggered and
>> there's a non-zero cost to that. What do you think?
>>
>> Cheers,
>>
>> Chris
>>
>> On Fri, Oct 14, 2022 at 12:59 AM Ashwin
>> wrote:
>>
>> > Thanks for KIP Chris - I think this is a useful feature.
>> >
>> > Can you please elaborate on the following in the KIP -
>> >
>> > 1. How would the response of GET /connectors/{connector}/offsets look
>> like
>> > if the worker has both global and connector specific offsets topic ?
>> >
>> > 2. How can we pass the reset options like shift-by , to-date-time etc.
>> > using a REST API like DELETE /connectors/{connector}/offsets ?
>> >
>> > 3. Today PAUSE operation on a connector invokes its stop method - will
>> > there be a change here to reduce confusion with the new proposed STOPPED
>> > state ?
>> >
>> > Thanks,
>> > Ashwin
>> >
>> > On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I noticed a fairly large gap in the first version of this KIP that I
>> > > published last Friday, which has to do with accommodating connectors
>> > > that target different Kafka clusters than the one that the Kafka
>> Connect
>> > > cluster uses for its internal topics and source connectors with
>> dedicated
>> > > offsets topics. I've since updated the KIP to address this gap, which
>> has
>> > > substantially altered the design. Wanted to give a heads-up to anyone
>> > > that's already started reviewing.
>> > >
>> > > Cheers,
>> > >
>> > > Chris
>> > >
>> > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > I'd like to begin discussion on a KIP to add offsets support to the
>> > Kafka
>> > > > Connect REST API:
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Chris
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
(either explicitly, or implicitly via
> cluster downgrade). What do you think about adding a URL query parameter
> that allows users to dictate which view of the connector's offsets they are
> given in the REST response, with options for the worker's global topic, the
> connector-specific topic, and the combined view of them that the connector
> and its tasks see (which would be the default)? This may be too much for V1
> but it feels like it's at least worth exploring a bit.
>
> 2. There is no option for this at the moment. Reset semantics are extremely
> coarse-grained; for source connectors, we delete all source offsets, and
> for sink connectors, we delete the entire consumer group. I'm hoping this
> will be enough for V1 and that, if there's sufficient demand for it, we can
> introduce a richer API for resetting or even modifying connector offsets in
> a follow-up KIP.
>
> 3. Good eye :) I think it's fine to keep the existing behavior for the
> PAUSED state with the Connector instance, since the primary purpose of the
> Connector is to generate task configs and monitor the external system for
> changes. If there's no chance for tasks to be running anyways, I don't see
> much value in allowing paused connectors to generate new task configs,
> especially since each time that happens a rebalance is triggered and
> there's a non-zero cost to that. What do you think?
>
> Cheers,
>
> Chris
>
> On Fri, Oct 14, 2022 at 12:59 AM Ashwin
> wrote:
>
> > Thanks for KIP Chris - I think this is a useful feature.
> >
> > Can you please elaborate on the following in the KIP -
> >
> > 1. How would the response of GET /connectors/{connector}/offsets look
> like
> > if the worker has both global and connector specific offsets topic ?
> >
> > 2. How can we pass the reset options like shift-by , to-date-time etc.
> > using a REST API like DELETE /connectors/{connector}/offsets ?
> >
> > 3. Today PAUSE operation on a connector invokes its stop method - will
> > there be a change here to reduce confusion with the new proposed STOPPED
> > state ?
> >
> > Thanks,
> > Ashwin
> >
> > On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
> > wrote:
> >
> > > Hi all,
> > >
> > > I noticed a fairly large gap in the first version of this KIP that I
> > > published last Friday, which has to do with accommodating connectors
> > > that target different Kafka clusters than the one that the Kafka
> Connect
> > > cluster uses for its internal topics and source connectors with
> dedicated
> > > offsets topics. I've since updated the KIP to address this gap, which
> has
> > > substantially altered the design. Wanted to give a heads-up to anyone
> > > that's already started reviewing.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to begin discussion on a KIP to add offsets support to the
> > Kafka
> > > > Connect REST API:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
> >
>
Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
unexpected. > > 2. The idea is to only perform zombie fencing when we know that it is > necessary (which is a principle borrowed from KIP-618), so in this case, > we'll only do it in response to an offsets reset request, and not when a > connector is stopped. After being stopped, it's possible that the connector > gets deleted, in which case, a proactive round of fencing would have served > no benefit. It's also worth noting that publishing an empty set of task > configs is not the same as tombstoning existing task configs; putting a > connector into the STOPPED state should require no tombstones to be emitted > to the config topic. > > Cheers, > > Chris > > On Thu, Oct 13, 2022 at 6:26 PM Greg Harris > wrote: > > > Hey Chris, > > > > Thanks for the KIP! > > > > I think this is an important feature for both development and operations > > use-cases, and it's an obvious gap in the REST feature set. > > I also appreciate the incremental nature of the KIP and the future > > extensions that will now be possible. > > > > I had a couple of questions about the design and it's extensibility: > > > > 1. How do you imagine the API will behave with connectors that have > > extremely large numbers of partitions (thousands or more) and/or source > > connectors with large amounts of data per partition? > > > > 2. Does the new STOPPED state need any special integration with the > > rebalance subsystem, or can the rebalance algorithms remain ignorant of > the > > target state of connectors? > > > > And about the implementation: > > > > 1. For my own edification, what is the difference between deleting a > > consumer group and deleting all known offsets for that group? Does > deleting > > the group offer better/easier atomicity? > > > > 2. For EOS sources, will stopping the connector and tombstoning the task > > configs perform a fence-out, or will that fence-out only occur when > > performing the offsets DELETE operation? > > > > Thanks! > > Greg > > > > On 2022/10/13 20:52:26 Chris Egerton wrote: > > > Hi all, > > > > > > I noticed a fairly large gap in the first version of this KIP that I > > > published last Friday, which has to do with accommodating connectors > > > that target different Kafka clusters than the one that the Kafka > Connect > > > cluster uses for its internal topics and source connectors with > dedicated > > > offsets topics. I've since updated the KIP to address this gap, which > has > > > substantially altered the design. Wanted to give a heads-up to anyone > > > that's already started reviewing. > > > > > > Cheers, > > > > > > Chris > > > > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > > > > > > Hi all, > > > > > > > > I'd like to begin discussion on a KIP to add offsets support to the > > Kafka > > > > Connect REST API: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi Ashwin,
Thanks for your thoughts. Regarding your questions:
1. The response would show the offsets that are visible to the source
connector, so it would combine the contents of the two topics, giving
priority to offsets present in the connector-specific topic. I'm imagining
a follow-up question that some people may have in response to that is
whether we'd want to provide insight into the contents of a single topic at
a time. It may be useful to be able to see this information in order to
debug connector issues or verify that it's safe to stop using a
connector-specific offsets topic (either explicitly, or implicitly via
cluster downgrade). What do you think about adding a URL query parameter
that allows users to dictate which view of the connector's offsets they are
given in the REST response, with options for the worker's global topic, the
connector-specific topic, and the combined view of them that the connector
and its tasks see (which would be the default)? This may be too much for V1
but it feels like it's at least worth exploring a bit.
2. There is no option for this at the moment. Reset semantics are extremely
coarse-grained; for source connectors, we delete all source offsets, and
for sink connectors, we delete the entire consumer group. I'm hoping this
will be enough for V1 and that, if there's sufficient demand for it, we can
introduce a richer API for resetting or even modifying connector offsets in
a follow-up KIP.
3. Good eye :) I think it's fine to keep the existing behavior for the
PAUSED state with the Connector instance, since the primary purpose of the
Connector is to generate task configs and monitor the external system for
changes. If there's no chance for tasks to be running anyways, I don't see
much value in allowing paused connectors to generate new task configs,
especially since each time that happens a rebalance is triggered and
there's a non-zero cost to that. What do you think?
Cheers,
Chris
On Fri, Oct 14, 2022 at 12:59 AM Ashwin
wrote:
> Thanks for KIP Chris - I think this is a useful feature.
>
> Can you please elaborate on the following in the KIP -
>
> 1. How would the response of GET /connectors/{connector}/offsets look like
> if the worker has both global and connector specific offsets topic ?
>
> 2. How can we pass the reset options like shift-by , to-date-time etc.
> using a REST API like DELETE /connectors/{connector}/offsets ?
>
> 3. Today PAUSE operation on a connector invokes its stop method - will
> there be a change here to reduce confusion with the new proposed STOPPED
> state ?
>
> Thanks,
> Ashwin
>
> On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
> wrote:
>
> > Hi all,
> >
> > I noticed a fairly large gap in the first version of this KIP that I
> > published last Friday, which has to do with accommodating connectors
> > that target different Kafka clusters than the one that the Kafka Connect
> > cluster uses for its internal topics and source connectors with dedicated
> > offsets topics. I've since updated the KIP to address this gap, which has
> > substantially altered the design. Wanted to give a heads-up to anyone
> > that's already started reviewing.
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote:
> >
> > > Hi all,
> > >
> > > I'd like to begin discussion on a KIP to add offsets support to the
> Kafka
> > > Connect REST API:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> >
>
Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
design. Wanted to give a heads-up to anyone > > that's already started reviewing. > > > > Cheers, > > > > Chris > > > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > > > > Hi all, > > > > > > I'd like to begin discussion on a KIP to add offsets support to the > Kafka > > > Connect REST API: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > > > Cheers, > > > > > > Chris > > > > > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Thanks for KIP Chris - I think this is a useful feature.
Can you please elaborate on the following in the KIP -
1. How would the response of GET /connectors/{connector}/offsets look like
if the worker has both global and connector specific offsets topic ?
2. How can we pass the reset options like shift-by , to-date-time etc.
using a REST API like DELETE /connectors/{connector}/offsets ?
3. Today PAUSE operation on a connector invokes its stop method - will
there be a change here to reduce confusion with the new proposed STOPPED
state ?
Thanks,
Ashwin
On Fri, Oct 14, 2022 at 2:22 AM Chris Egerton
wrote:
> Hi all,
>
> I noticed a fairly large gap in the first version of this KIP that I
> published last Friday, which has to do with accommodating connectors
> that target different Kafka clusters than the one that the Kafka Connect
> cluster uses for its internal topics and source connectors with dedicated
> offsets topics. I've since updated the KIP to address this gap, which has
> substantially altered the design. Wanted to give a heads-up to anyone
> that's already started reviewing.
>
> Cheers,
>
> Chris
>
> On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote:
>
> > Hi all,
> >
> > I'd like to begin discussion on a KIP to add offsets support to the Kafka
> > Connect REST API:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> >
> > Cheers,
> >
> > Chris
> >
>
RE: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hey Chris, Thanks for the KIP! I think this is an important feature for both development and operations use-cases, and it's an obvious gap in the REST feature set. I also appreciate the incremental nature of the KIP and the future extensions that will now be possible. I had a couple of questions about the design and it's extensibility: 1. How do you imagine the API will behave with connectors that have extremely large numbers of partitions (thousands or more) and/or source connectors with large amounts of data per partition? 2. Does the new STOPPED state need any special integration with the rebalance subsystem, or can the rebalance algorithms remain ignorant of the target state of connectors? And about the implementation: 1. For my own edification, what is the difference between deleting a consumer group and deleting all known offsets for that group? Does deleting the group offer better/easier atomicity? 2. For EOS sources, will stopping the connector and tombstoning the task configs perform a fence-out, or will that fence-out only occur when performing the offsets DELETE operation? Thanks! Greg On 2022/10/13 20:52:26 Chris Egerton wrote: > Hi all, > > I noticed a fairly large gap in the first version of this KIP that I > published last Friday, which has to do with accommodating connectors > that target different Kafka clusters than the one that the Kafka Connect > cluster uses for its internal topics and source connectors with dedicated > offsets topics. I've since updated the KIP to address this gap, which has > substantially altered the design. Wanted to give a heads-up to anyone > that's already started reviewing. > > Cheers, > > Chris > > On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > > > Hi all, > > > > I'd like to begin discussion on a KIP to add offsets support to the Kafka > > Connect REST API: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > > > Cheers, > > > > Chris > > >
Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi all, I noticed a fairly large gap in the first version of this KIP that I published last Friday, which has to do with accommodating connectors that target different Kafka clusters than the one that the Kafka Connect cluster uses for its internal topics and source connectors with dedicated offsets topics. I've since updated the KIP to address this gap, which has substantially altered the design. Wanted to give a heads-up to anyone that's already started reviewing. Cheers, Chris On Fri, Oct 7, 2022 at 1:29 PM Chris Egerton wrote: > Hi all, > > I'd like to begin discussion on a KIP to add offsets support to the Kafka > Connect REST API: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect > > Cheers, > > Chris >
[DISCUSS] KIP-875: First-class offsets support in Kafka Connect
Hi all, I'd like to begin discussion on a KIP to add offsets support to the Kafka Connect REST API: https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect Cheers, Chris
[jira] [Resolved] (KAFKA-10759) ARM support for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-10759. Fix Version/s: 3.0.0 Resolution: Fixed > ARM support for Kafka > - > > Key: KAFKA-10759 > URL: https://issues.apache.org/jira/browse/KAFKA-10759 > Project: Kafka > Issue Type: Improvement > Components: build >Reporter: PengLei >Priority: Major > Fix For: 3.0.0 > > Attachments: build_output.log, run_test_output.log > > > ARM support for Kafka. > I tried to deploy the Kafka cluster on the ARM server, but unfortunately I > did not find the official ARM release for Kafka. I think more and more > people will try the same thing as I do. > Now the CI of kafka (in github) is handled by jenkins-ci. While the test is > running under x86 ARCH, the arm ARCH is missing. This leads an problem that > we don't have a way to test every pull request that if it'll break the kafka > deployment on arm or not. Similarly, we cannot provide the ARM release > package without the ARM CI. > If Apache Kafka community has interested with it, I can help for the > integration. > This is the umbrella issue to track the efforts to make Kafka run on ARM > processors. > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (KAFKA-13671) Power (ppc64le) support for kafka
[ https://issues.apache.org/jira/browse/KAFKA-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-13671. Fix Version/s: 3.2.0 Resolution: Fixed > Power (ppc64le) support for kafka > - > > Key: KAFKA-13671 > URL: https://issues.apache.org/jira/browse/KAFKA-13671 > Project: Kafka > Issue Type: Improvement > Components: build >Reporter: Abhijit >Assignee: Mickael Maison >Priority: Major > Fix For: 3.2.0 > > Attachments: kafka_ConsumerBounceTest_IT.txt, kafka_IT.txt, > kafka_UnitTest.txt > > > Support for Power architecture (ppc64le) for apache kafka. > What is IBM Power architecture? > It is a RISC architecture and IBM has recently made its ISA (Instruction Set > Architecture) opensource and in doing so, they have significantly contributed > back to the opensource community at large. Many of the pioneers of banking > and HPC industries today run on ppc64le architecture. > As an ongoing effort to enable open-source projects where Power architecture > can add value, we are trying to enable kafka on Power. > IBM has already donated a ppc64le ubuntu VM to the community to be added to > the jenkins cluster for Apache Kafka builds. The VM spec has been reviewed in > the community and deemed suitable for kafka builds. > Jira: https://issues.apache.org/jira/browse/INFRA-22612 > I can help with the work needed to add new jenkins job for Power. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (KAFKA-13671) Power (ppc64le) support for kafka
Abhijit created KAFKA-13671: --- Summary: Power (ppc64le) support for kafka Key: KAFKA-13671 URL: https://issues.apache.org/jira/browse/KAFKA-13671 Project: Kafka Issue Type: Improvement Components: build Reporter: Abhijit Support for Power architecture (ppc64le) for apache kafka. What is IBM Power architecture? It is a RISC architecture and IBM has recently made its ISA (Instruction Set Architecture) opensource and in doing so, they have significantly contributed back to the opensource community at large. Many of the pioneers of banking and HPC industries today run on ppc64le architecture. As an ongoing effort to enable open-source projects where Power architecture can add value, we are trying to enable kafka on Power. IBM has already donated a ppc64le ubuntu VM to the community to be added to the jenkins cluster for Apache Kafka builds. The VM spec has been reviews in the community and deemed suitable for kafka builds. Jira: https://issues.apache.org/jira/browse/INFRA-22612 I can help with the work needed to add new jenkins job for Power. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options
t; inline.
> >
> > Am Di., 7. Dez. 2021 um 23:49 Uhr schrieb Chris Egerton
> > :
> >
> > > Hi Gunnar,
> > >
> > > Thanks for the KIP! The section on backwards compatibility is
> especially
> > > impressive and was enjoyable to read.
> > >
> >
> > Excellent, that's great to hear!
> >
> > Overall I like the direction of the KIP (and in fact just ran into a
> > > situation yesterday where it would be valuable). I only have one major
> > > thought: could we add similar validate methods for the Converter and
> > > HeaderConverter interfaces? With KIP-769 [1], it looks like we'll have
> a
> > > new Converter::config method, so if that makes it through, it should
> be a
> > > matter of just adding the same methods to those interfaces as well
> > > (although we may want to be tolerant of null ConfigDef objects being
> > > returned from HeaderConverter::config since the Connect framework has
> not
> > > been enforcing this requirement to date).
> > >
> >
> > Yes, I think it's a good idea to expand the scope of the KIP to cover all
> > these contracts. I have updated the KIP document accordingly.
> >
> > >
> > > That aside, a few small nits:
> > >
> > > 1. The "This page is meant as a template" section can be removed :)
> > > 2. The "Current Status" can be updated to "Under Discussion"
> > > 3. Might want to add javadocs to the newly-proposed validate method
> (I'm
> > > assuming they'll largely mirror the ones for the existing
> > > Connector::validate method, but we may also want to add a {@since} tag
> or
> > > some other information on which versions of Connect will leverage the
> > > method).
> > >
> >
> > Done.
> >
> > I will try and create a PR for this work in January next year.
> >
> > All the best,
> >
> > --Gunnar
> >
> > [1] -
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+plugins+and+retrieve+their+configuration+definitions#KIP769:ConnectAPIstolistallpluginsandretrievetheirconfigurationdefinitions-PublicInterfaces
> > > (section labeled "Converter interface"
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Nov 24, 2021 at 11:32 AM Gunnar Morling
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > I would like to propose a KIP for Apache Kafka Connect which adds
> > > > validation support for SMT-related configuration options:
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > >
> > > > This feature allows users to make sure an SMT is configured correctly
> > > > before actually putting a connector with that SMT in place.
> > > >
> > > > Any feedback, comments, and suggestions around this proposal will
> > > > be greatly appreciated.
> > > >
> > > > Thanks,
> > > >
> > > > --Gunnar
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options
Hi Gunnar,
Thanks, this looks great. I'm ready to cast a non-binding on the vote
thread when it comes.
One small non-blocking nit: I like that you call out that the new
validation steps will take place when a connector gets registered or
updated. IMO this is important enough to be included in the "Public
Interfaces" section as that type of preflight check is arguably more
important than the PUT /connector-plugins/{name}/config/validate endpoint,
when considering that use of the validation endpoint is strictly opt-in,
but preflight checks for new connector configs are unavoidable (without
resorting to devious hacks like publishing directly to the config topic).
But this is really minor, I'm happy to +1 the KIP as-is.
Cheers,
Chris
On Tue, Dec 21, 2021 at 8:43 AM Gunnar Morling
wrote:
> Hey Chris,
>
> Thanks a lot for reviewing this KIP and your comments! Some more answers
> inline.
>
> Am Di., 7. Dez. 2021 um 23:49 Uhr schrieb Chris Egerton
> :
>
> > Hi Gunnar,
> >
> > Thanks for the KIP! The section on backwards compatibility is especially
> > impressive and was enjoyable to read.
> >
>
> Excellent, that's great to hear!
>
> Overall I like the direction of the KIP (and in fact just ran into a
> > situation yesterday where it would be valuable). I only have one major
> > thought: could we add similar validate methods for the Converter and
> > HeaderConverter interfaces? With KIP-769 [1], it looks like we'll have a
> > new Converter::config method, so if that makes it through, it should be a
> > matter of just adding the same methods to those interfaces as well
> > (although we may want to be tolerant of null ConfigDef objects being
> > returned from HeaderConverter::config since the Connect framework has not
> > been enforcing this requirement to date).
> >
>
> Yes, I think it's a good idea to expand the scope of the KIP to cover all
> these contracts. I have updated the KIP document accordingly.
>
> >
> > That aside, a few small nits:
> >
> > 1. The "This page is meant as a template" section can be removed :)
> > 2. The "Current Status" can be updated to "Under Discussion"
> > 3. Might want to add javadocs to the newly-proposed validate method (I'm
> > assuming they'll largely mirror the ones for the existing
> > Connector::validate method, but we may also want to add a {@since} tag or
> > some other information on which versions of Connect will leverage the
> > method).
> >
>
> Done.
>
> I will try and create a PR for this work in January next year.
>
> All the best,
>
> --Gunnar
>
> [1] -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+plugins+and+retrieve+their+configuration+definitions#KIP769:ConnectAPIstolistallpluginsandretrievetheirconfigurationdefinitions-PublicInterfaces
> > (section labeled "Converter interface"
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Nov 24, 2021 at 11:32 AM Gunnar Morling
> > wrote:
> >
> > > Hey all,
> > >
> > > I would like to propose a KIP for Apache Kafka Connect which adds
> > > validation support for SMT-related configuration options:
> > >
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > >
> > > This feature allows users to make sure an SMT is configured correctly
> > > before actually putting a connector with that SMT in place.
> > >
> > > Any feedback, comments, and suggestions around this proposal will
> > > be greatly appreciated.
> > >
> > > Thanks,
> > >
> > > --Gunnar
> > >
> >
>
Re: [DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options
Hey Chris,
Thanks a lot for reviewing this KIP and your comments! Some more answers
inline.
Am Di., 7. Dez. 2021 um 23:49 Uhr schrieb Chris Egerton
:
> Hi Gunnar,
>
> Thanks for the KIP! The section on backwards compatibility is especially
> impressive and was enjoyable to read.
>
Excellent, that's great to hear!
Overall I like the direction of the KIP (and in fact just ran into a
> situation yesterday where it would be valuable). I only have one major
> thought: could we add similar validate methods for the Converter and
> HeaderConverter interfaces? With KIP-769 [1], it looks like we'll have a
> new Converter::config method, so if that makes it through, it should be a
> matter of just adding the same methods to those interfaces as well
> (although we may want to be tolerant of null ConfigDef objects being
> returned from HeaderConverter::config since the Connect framework has not
> been enforcing this requirement to date).
>
Yes, I think it's a good idea to expand the scope of the KIP to cover all
these contracts. I have updated the KIP document accordingly.
>
> That aside, a few small nits:
>
> 1. The "This page is meant as a template" section can be removed :)
> 2. The "Current Status" can be updated to "Under Discussion"
> 3. Might want to add javadocs to the newly-proposed validate method (I'm
> assuming they'll largely mirror the ones for the existing
> Connector::validate method, but we may also want to add a {@since} tag or
> some other information on which versions of Connect will leverage the
> method).
>
Done.
I will try and create a PR for this work in January next year.
All the best,
--Gunnar
[1] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+plugins+and+retrieve+their+configuration+definitions#KIP769:ConnectAPIstolistallpluginsandretrievetheirconfigurationdefinitions-PublicInterfaces
> (section labeled "Converter interface"
>
> Cheers,
>
> Chris
>
> On Wed, Nov 24, 2021 at 11:32 AM Gunnar Morling
> wrote:
>
> > Hey all,
> >
> > I would like to propose a KIP for Apache Kafka Connect which adds
> > validation support for SMT-related configuration options:
> >
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> >
> > This feature allows users to make sure an SMT is configured correctly
> > before actually putting a connector with that SMT in place.
> >
> > Any feedback, comments, and suggestions around this proposal will
> > be greatly appreciated.
> >
> > Thanks,
> >
> > --Gunnar
> >
>
Re: [DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options
Hi Gunnar,
Thanks for the KIP! The section on backwards compatibility is especially
impressive and was enjoyable to read.
Overall I like the direction of the KIP (and in fact just ran into a
situation yesterday where it would be valuable). I only have one major
thought: could we add similar validate methods for the Converter and
HeaderConverter interfaces? With KIP-769 [1], it looks like we'll have a
new Converter::config method, so if that makes it through, it should be a
matter of just adding the same methods to those interfaces as well
(although we may want to be tolerant of null ConfigDef objects being
returned from HeaderConverter::config since the Connect framework has not
been enforcing this requirement to date).
That aside, a few small nits:
1. The "This page is meant as a template" section can be removed :)
2. The "Current Status" can be updated to "Under Discussion"
3. Might want to add javadocs to the newly-proposed validate method (I'm
assuming they'll largely mirror the ones for the existing
Connector::validate method, but we may also want to add a {@since} tag or
some other information on which versions of Connect will leverage the
method).
[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+plugins+and+retrieve+their+configuration+definitions#KIP769:ConnectAPIstolistallpluginsandretrievetheirconfigurationdefinitions-PublicInterfaces
(section labeled "Converter interface"
Cheers,
Chris
On Wed, Nov 24, 2021 at 11:32 AM Gunnar Morling
wrote:
> Hey all,
>
> I would like to propose a KIP for Apache Kafka Connect which adds
> validation support for SMT-related configuration options:
>
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
>
> This feature allows users to make sure an SMT is configured correctly
> before actually putting a connector with that SMT in place.
>
> Any feedback, comments, and suggestions around this proposal will
> be greatly appreciated.
>
> Thanks,
>
> --Gunnar
>
[DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options
Hey all, I would like to propose a KIP for Apache Kafka Connect which adds validation support for SMT-related configuration options: https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options This feature allows users to make sure an SMT is configured correctly before actually putting a connector with that SMT in place. Any feedback, comments, and suggestions around this proposal will be greatly appreciated. Thanks, --Gunnar
[jira] [Created] (KAFKA-13478) KIP-802: Validation Support for Kafka Connect SMT Options
Gunnar Morling created KAFKA-13478: -- Summary: KIP-802: Validation Support for Kafka Connect SMT Options Key: KAFKA-13478 URL: https://issues.apache.org/jira/browse/KAFKA-13478 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Gunnar Morling Implement [KIP-802|KIP-802: Validation Support for Kafka Connect SMT Options], adding validation support for SMT options. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [kafka-site] mjsax merged pull request #376: MINOR: remove note on future multi-cluster support of Kafka Streams
mjsax merged pull request #376: URL: https://github.com/apache/kafka-site/pull/376 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka-site] mjsax commented on pull request #376: MINOR: remove note on future multi-cluster support of Kafka Streams
mjsax commented on pull request #376: URL: https://github.com/apache/kafka-site/pull/376#issuecomment-930596904 Good catch! Wrong `Tip:` section Let me fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka-site] mjsax commented on pull request #376: MINOR: remove note on future multi-cluster support of Kafka Streams
mjsax commented on pull request #376: URL: https://github.com/apache/kafka-site/pull/376#issuecomment-930596904 Good catch! Wrong `Tip:` section Let me fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka-site] mjsax opened a new pull request #376: MINOR: remove note on future multi-cluster support of Kafka Streams
mjsax opened a new pull request #376: URL: https://github.com/apache/kafka-site/pull/376 Follow up to https://github.com/apache/kafka/pull/11363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: ARM support for Kafka
Hi Kafka The email about whether Kafka supports arm ci has been send for a while and no objections were raised. Can we just keep moving forward ? @junrao <http://twitter.com/junrao> @ijuma <https://twitter.com/ijuma> , The PR I have submitted [https://github.com/apache/kafka/pull/9872]. I don't have permission to configure it to work. Can you help me? Thanks Lei Peng Peng Lei 于2021年1月18日周一 上午10:43写道： > Hi Kafka > I'm pushing kafka to support arm ci. I think this is a very > meaningful work because of it helps kafka get better. If Kafka supports > arm, the following benefits are provided: > 1. Kafka can be easily used by arm users. They do not need to worry > about kafka being unavailable on the arm platform. With the arm CI, we can > ensure that each code integration can run properly on the arm platform. If > we do more, we can provide the official arm release package to benefit the > ARM users. I think this is a very meaningful work for Kafka users on the > arm platform. > 2. More and more big data components, including Spark, Flink, and > Hadoop, are supported on the arm platform. As the most popular message > middleware, Kafka is often integrated with other components. If the arm > platform is supported, the deployment of the entire service is more > convenient and quick.In [https://ci-hadoop.apache.org/], you can see > there are two arm servers to support arm ci for hadoop. > 3. It is not difficult for kafka to support arm because of jvm. So > it's not a waste of energy. > > In order to get kafka to support arm ci, I did the following: > 1、[https://issues.apache.org/jira/browse/KAFKA-10759],All test cases > are tested on the arm platform. > 2、[https://issues.apache.org/jira/browse/INFRA-21203],Provide the arm > machine to the Infra for testing the arm CI of the Kafka. > 3、[https://github.com/apache/kafka/pull/9872], Modify the jenkinsfile > to support the arm ci for kafka. > > @junrao <http://twitter.com/junrao> @nehanarkhede > <http://twitter.com/nehanarkhede> @allthingshadoop > <https://twitter.com/allthingshadoop> @jaykreps > <http://twitter.com/jaykreps> @mumrah <https://twitter.com/mumrah> > @sriramsub1 <http://twitter.com/sriramsub1> @blueboxtraveler > <http://twitter.com/BlueBoxTraveler> @guozhangwang > <https://twitter.com/guozhangwang> @gwenshap > <https://twitter.com/gwenshap> @d3fmacro <https://twitter.com/d3fmacro> > @ewencp <https://twitter.com/ewencp> @ijuma <https://twitter.com/ijuma> > @gchenke <https://twitter.com/gchenke> @damianguy > <https://twitter.com/damianguy> @MatthiasJSax > <https://twitter.com/MatthiasJSax> @vahidh <https://twitter.com/vahidh> > @rhauch <https://twitter.com/rhauch> @davidjacot > <https://twitter.com/davidjacot> What do you think about kafka's support > for arm ci? > > Thanks > Lei Peng > >
ARM support for Kafka
Hi Kafka I'm pushing kafka to support arm ci. I think this is a very meaningful work because of it helps kafka get better. If Kafka supports arm, the following benefits are provided: 1. Kafka can be easily used by arm users. They do not need to worry about kafka being unavailable on the arm platform. With the arm CI, we can ensure that each code integration can run properly on the arm platform. If we do more, we can provide the official arm release package to benefit the ARM users. I think this is a very meaningful work for Kafka users on the arm platform. 2. More and more big data components, including Spark, Flink, and Hadoop, are supported on the arm platform. As the most popular message middleware, Kafka is often integrated with other components. If the arm platform is supported, the deployment of the entire service is more convenient and quick.In [https://ci-hadoop.apache.org/], you can see there are two arm servers to support arm ci for hadoop. 3. It is not difficult for kafka to support arm because of jvm. So it's not a waste of energy. In order to get kafka to support arm ci, I did the following: 1、[https://issues.apache.org/jira/browse/KAFKA-10759],All test cases are tested on the arm platform. 2、[https://issues.apache.org/jira/browse/INFRA-21203],Provide the arm machine to the Infra for testing the arm CI of the Kafka. 3、[https://github.com/apache/kafka/pull/9872], Modify the jenkinsfile to support the arm ci for kafka. @junrao <http://twitter.com/junrao> @nehanarkhede <http://twitter.com/nehanarkhede> @allthingshadoop <https://twitter.com/allthingshadoop> @jaykreps <http://twitter.com/jaykreps> @mumrah <https://twitter.com/mumrah> @sriramsub1 <http://twitter.com/sriramsub1> @blueboxtraveler <http://twitter.com/BlueBoxTraveler> @guozhangwang <https://twitter.com/guozhangwang> @gwenshap <https://twitter.com/gwenshap> @d3fmacro <https://twitter.com/d3fmacro> @ewencp <https://twitter.com/ewencp> @ijuma <https://twitter.com/ijuma> @gchenke <https://twitter.com/gchenke> @damianguy <https://twitter.com/damianguy> @MatthiasJSax <https://twitter.com/MatthiasJSax> @vahidh <https://twitter.com/vahidh> @rhauch <https://twitter.com/rhauch> @davidjacot <https://twitter.com/davidjacot> What do you think about kafka's support for arm ci? Thanks Lei Peng
[jira] [Created] (KAFKA-10759) ARM support for Kafka
PengLei created KAFKA-10759: --- Summary: ARM support for Kafka Key: KAFKA-10759 URL: https://issues.apache.org/jira/browse/KAFKA-10759 Project: Kafka Issue Type: Improvement Components: build Reporter: PengLei ARM support for Kafka. I tried to deploy the Kafka cluster on the ARM server, but unfortunately I did not find the official ARM release for Kafka. I think more and more people will try the same thing as I do. Now the CI of kafka (in github) is handled by jenkins-ci. While the test is running under x86 ARCH, the arm ARCH is missing. This leads an problem that we don't have a way to test every pull request that if it'll break the kafka deployment on arm or not. Similarly, we cannot provide the ARM release package without the ARM CI. If Apache Kafka community has interested with it, I can help for the integration. This is the umbrella issue to track the efforts to make Kafka run on ARM processors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
