Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

I just wanted to bump this and see if anyone had more feedback before
trying to call a vote for this for 4.0?

Chris

On Mon, Apr 1, 2024 at 3:41 PM Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Greg,
>
> 1. Ok sounds good we can target JDK 17 in this KIP if we decide to do that.
> 2. For the EE version, I don't think it really matters since we won't be
> using any new features that I am aware of. It's just something I noticed
> that we will need to pick because Jetty 12 supports multiple versions so it
> would affect which spec jars we use.  In the past Jetty versions have been
> tied to a specific Servlet spec but the new Jetty 12 they have abstracted
> things away and they support multiple versions simultaneously. There's
> different versions for all the specs but the primary one to note for us
> would be that JavaEE 9 uses the Servlet 5.0 spec and JavaEE 10 uses the
> Servlet 6.0 spec. JavaEE 11 is under development and will use the Servlet
> 6.1 spec. So we may not really need to call out the EE version at all if it
> doesn't matter and we are not using specific features but I wanted to bring
> it up since multiple versions are listed as being compatible with Jetty 12
> so we need to pick one. On the main page they list the different servlet
> specs they support: https://eclipse.dev/jetty/
> 3. Right, I didn't mean we should include it in the KIP, I was more asking
> I guess how to go about things. It looks like we could use a lot of it and
> adapt the work already done. While it's under the Apache 2.0 license and we
> could use it, someone else wrote it so it would still be good to properly
> credit that person as you mentioned. If I work on it I would probably start
> over with a new branch and just use the old PR as a guide and then maybe
> figure out a way to credit the original author. There's always that
> co-author tag that could be used I think.
>
> Chris
>
>
> On Mon, Apr 1, 2024 at 12:11 PM Greg Harris 
> wrote:
>
>> Hey Chris,
>>
>> Thanks for your questions!
>>
>> 1. KIPs are generally immutable once they've been voted on. In this
>> case, I think it's better that this KIP include the bump to Java 17
>> just for Connect and MirrorMaker 2, and should include that in the KIP
>> title.
>> 2. I'm not familiar with the difference, can you provide some more
>> context that would help us make a decision? AFAIU we haven't specified
>> an EE version in the past, and we don't do any sort of automated
>> testing for compatibility. I think it would be good to call out which
>> components have JavaEE-sensitive dependencies (just connect-runtime?).
>> We do not want to accidentally give users the idea that the clients
>> depend on the JavaEE version, as that could be very confusing.
>> 3. That's an implementation detail left up to anyone that effects this
>> KIP on the repo, and doesn't need to be mentioned in the KIP itself. I
>> have seen people adopt changes from un-merged PRs after the original
>> contributor has lost interest, while still crediting the original
>> contributor for their portion of the changes. If you're doing this,
>> then it's ultimately up to your judgement.
>>
>> Thanks,
>> Greg
>>
>> On Mon, Apr 1, 2024 at 6:30 AM Christopher Shannon
>>  wrote:
>> >
>> > Hi Greg,
>> >
>> > Thanks for the detailed analysis on the connect framework. It sounds
>> like
>> > hopefully we can go ahead and require JDK 17+ and bump that dependency
>> for
>> > the ConnectRestExtensionContext.
>> >
>> > I agree we can leave it open and hear what others think as well about
>> the
>> > requirement, if everyone ends up agreeing, I can update the KIP to
>> reflect
>> > requiring JDK 17 and going with Jetty 12.
>> >
>> > I had a couple of questions
>> > 1) If we go with JDK 17 as a requirement for the Connect framework,
>> should
>> > we modify and make that part of KIP-1013 or keep it in this one?
>> > 2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
>> > matters in this case.
>> > 3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
>> > starting point or maybe we can create a new PR and use it as a basis?
>> It's
>> > a bit old so I suspect there would be a ton of conflicts so it might be
>> > best to start over and use it as a guide. I can work on a new PR once we
>> > are all on the same page. I don't think it would take too long to put
>> > together since most of the work is just dependency updates and package
>> > renaming.
>> >
>

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

Greg,

1. Ok sounds good we can target JDK 17 in this KIP if we decide to do that.
2. For the EE version, I don't think it really matters since we won't be
using any new features that I am aware of. It's just something I noticed
that we will need to pick because Jetty 12 supports multiple versions so it
would affect which spec jars we use.  In the past Jetty versions have been
tied to a specific Servlet spec but the new Jetty 12 they have abstracted
things away and they support multiple versions simultaneously. There's
different versions for all the specs but the primary one to note for us
would be that JavaEE 9 uses the Servlet 5.0 spec and JavaEE 10 uses the
Servlet 6.0 spec. JavaEE 11 is under development and will use the Servlet
6.1 spec. So we may not really need to call out the EE version at all if it
doesn't matter and we are not using specific features but I wanted to bring
it up since multiple versions are listed as being compatible with Jetty 12
so we need to pick one. On the main page they list the different servlet
specs they support: https://eclipse.dev/jetty/
3. Right, I didn't mean we should include it in the KIP, I was more asking
I guess how to go about things. It looks like we could use a lot of it and
adapt the work already done. While it's under the Apache 2.0 license and we
could use it, someone else wrote it so it would still be good to properly
credit that person as you mentioned. If I work on it I would probably start
over with a new branch and just use the old PR as a guide and then maybe
figure out a way to credit the original author. There's always that
co-author tag that could be used I think.

Chris


On Mon, Apr 1, 2024 at 12:11 PM Greg Harris 
wrote:

> Hey Chris,
>
> Thanks for your questions!
>
> 1. KIPs are generally immutable once they've been voted on. In this
> case, I think it's better that this KIP include the bump to Java 17
> just for Connect and MirrorMaker 2, and should include that in the KIP
> title.
> 2. I'm not familiar with the difference, can you provide some more
> context that would help us make a decision? AFAIU we haven't specified
> an EE version in the past, and we don't do any sort of automated
> testing for compatibility. I think it would be good to call out which
> components have JavaEE-sensitive dependencies (just connect-runtime?).
> We do not want to accidentally give users the idea that the clients
> depend on the JavaEE version, as that could be very confusing.
> 3. That's an implementation detail left up to anyone that effects this
> KIP on the repo, and doesn't need to be mentioned in the KIP itself. I
> have seen people adopt changes from un-merged PRs after the original
> contributor has lost interest, while still crediting the original
> contributor for their portion of the changes. If you're doing this,
> then it's ultimately up to your judgement.
>
> Thanks,
> Greg
>
> On Mon, Apr 1, 2024 at 6:30 AM Christopher Shannon
>  wrote:
> >
> > Hi Greg,
> >
> > Thanks for the detailed analysis on the connect framework. It sounds like
> > hopefully we can go ahead and require JDK 17+ and bump that dependency
> for
> > the ConnectRestExtensionContext.
> >
> > I agree we can leave it open and hear what others think as well about the
> > requirement, if everyone ends up agreeing, I can update the KIP to
> reflect
> > requiring JDK 17 and going with Jetty 12.
> >
> > I had a couple of questions
> > 1) If we go with JDK 17 as a requirement for the Connect framework,
> should
> > we modify and make that part of KIP-1013 or keep it in this one?
> > 2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
> > matters in this case.
> > 3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
> > starting point or maybe we can create a new PR and use it as a basis?
> It's
> > a bit old so I suspect there would be a ton of conflicts so it might be
> > best to start over and use it as a guide. I can work on a new PR once we
> > are all on the same page. I don't think it would take too long to put
> > together since most of the work is just dependency updates and package
> > renaming.
> >
> > Chris
> >
> >
> > On Fri, Mar 29, 2024 at 8:59 PM Greg Harris  >
> > wrote:
> >
> > > Hey all,
> > >
> > > I looked into how Debezium handled the javax->jakarta changeover for
> > > Quarkus, and found this release note:
> > >
> > >
> https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-quarkus-3
> > > It appears that Debezium 2.1 required Quarkus < 3.0, and Debezium 2.2
> > > required Quarkus >= 3.0. The upgrade for Kafka could be very similar
> > > and not 

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

Hi Greg,

Thanks for the detailed analysis on the connect framework. It sounds like
hopefully we can go ahead and require JDK 17+ and bump that dependency for
the ConnectRestExtensionContext.

I agree we can leave it open and hear what others think as well about the
requirement, if everyone ends up agreeing, I can update the KIP to reflect
requiring JDK 17 and going with Jetty 12.

I had a couple of questions
1) If we go with JDK 17 as a requirement for the Connect framework, should
we modify and make that part of KIP-1013 or keep it in this one?
2) Should we go with JavaEE 9 or JavaEE 10? I'm not sure how much it
matters in this case.
3) Can we just re-open https://github.com/apache/kafka/pull/10176 as a
starting point or maybe we can create a new PR and use it as a basis? It's
a bit old so I suspect there would be a ton of conflicts so it might be
best to start over and use it as a guide. I can work on a new PR once we
are all on the same page. I don't think it would take too long to put
together since most of the work is just dependency updates and package
renaming.

Chris


On Fri, Mar 29, 2024 at 8:59 PM Greg Harris 
wrote:

> Hey all,
>
> I looked into how Debezium handled the javax->jakarta changeover for
> Quarkus, and found this release note:
>
> https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-quarkus-3
> It appears that Debezium 2.1 required Quarkus < 3.0, and Debezium 2.2
> required Quarkus >= 3.0. The upgrade for Kafka could be very similar
> and not incur a major version release.
>
>  We can leave some time to hear from anyone else that is impacted by
> this change, but from the open source projects present on github, this
> LGTM.
>
> Thanks,
> Greg
>
> On Fri, Mar 29, 2024 at 5:27 PM Greg Harris  wrote:
> >
> > Hi Chris,
> >
> > Thank you so much for opening this KIP, and making sure Kafka keeps up
> > with the rest of the Java ecosystem!
> >
> > I took a look around at some Open Source connector implementations,
> > and checked their Java version support:
> > * The Aiven connect plugins (http, bigquery, jdbc, elasticsearch,
> > opensearch, commons, s3, transforms, gcs), 6/9 are tested with JDK 17
> > in CI, 2/9 JDK 11, and 1/9 JDK 8. I'll look into improving the testing
> > matrix, but I don't expect substantial problems with requiring JDK 17.
> > * The Debezium Project lists Java 11+ compatibility:
> > https://debezium.io/releases/ and appears to use Java 22 (ga) and 23
> > (ea) in their CI:
> >
> https://github.com/debezium/debezium/blob/9cdaa38453c9f065c6075d31636592a5b147518f/.github/workflows/jdk-outreach-workflow.yml#L20
> >
> > I think the bigger problem really is the ConnectRestExtension, since
> > we've baked the rs-api type into the signature of
> > ConnectRestExtensionContext.
> > * Aiven doesn't have any ConnectRestExtensions, so this isn't a concern
> for us.
> > * The Debezium Project has at least 6 ConnectRestExtension
> > implementations:
> >
> https://github.com/search?q=repo%3Adebezium%2Fdebezium+ConnectRestExtension+language%3AJava=code=Java
> > . Some of these are baked into artifacts that I know for a fact are
> > used in normal connect deployments.
> > * I found a healthcheck extension that looks unmaintained:
> >
> https://github.com/LoObp4ck/kafka-connect-healthchecks/blob/2d9dbfee900d9f85e6acd9a09bd04969afa46261/src/main/java/com/loobpack/data/kafka/connect/healthcheck/extension/HealthCheckConnectRestExtension.java#L9
> >
> > I figure that adopting this KIP would mean that the Debezium project
> > would be forced to bump their major version 3.0 to be compatible with
> > Connect 4.0, or otherwise change their packaging, so I'd like to hear
> > from the Debezium folks what they think of this proposal.
> >
> > Thanks,
> > Greg
> >
> > On Wed, Mar 27, 2024 at 4:43 PM Christopher Shannon
> >  wrote:
> > >
> > > Hi,
> > >
> > > I'm proposing a KIP for Kafka 4.0 to upgrade to to Jakarta and JavaEE 9
> > > APIs. This will also upgrade dependencies like Jetty and move away from
> > > the depcrated javax namespace to be in line with other libraries and
> > > frameworks. There was some initial
> > > <https://lists.apache.org/thread/p4qbqh8r77h4khn3yoof2b0gbq3wbc5q>
> > > discussion and below is the KIP.
> > >
> > > Please take a look and let me know what you think:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1032%3A+Upgrade+to+Jakarta+and+JavaEE+9+in+Kafka+4.0
> > >
> > > Thanks,
> > > Chris
>

[DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

[Christopher Shannon]

Hi,

I'm proposing a KIP for Kafka 4.0 to upgrade to to Jakarta and JavaEE 9
APIs. This will also upgrade dependencies like Jetty and move away from
the depcrated javax namespace to be in line with other libraries and
frameworks. There was some initial

discussion and below is the KIP.

Please take a look and let me know what you think:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1032%3A+Upgrade+to+Jakarta+and+JavaEE+9+in+Kafka+4.0

Thanks,
Chris

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon]

Hi Ismael,

Thanks for the feedback, I can definitely raise a KIP, that is no problem.

I will write one up and then we can have further discussion on the details.
I should have time to get one created later today or by tomorrow.

Chris


On Wed, Mar 27, 2024 at 11:54 AM Ismael Juma  wrote:

> Hi Christopher,
>
> Thanks for raising this. Moving to the new namespace makes sense - would
> you be willing to submit a KIP? The point you raised regarding Jetty 11 EOL
> and Jetty 12 requiring Java 17 is a good one and is worth discussing the
> trade-offs in more detail. I originally did not propose moving Connect to
> Java 17 because of the risk that it might break several connectors. If
> someone summarized the number of connectors that support Java 17 and the
> number that does not, it would be a useful first step in the discussion.
>
> Ismael
>
> On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Hi Greg,
> >
> > You are right that KIP-1013 and the JDK 17 upgrade is not directly
> relevant
> > because JDK 11 can still be used with Jakarta APIs. However, it's still
> > somewhat relevant and important because if we are stuck at JDK 11 then we
> > can't upgrade to certain versions. For Connect, there is a Jetty server
> for
> > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> > problem with using Jetty 11.x is that it is already EOL.
> >
> > So are we really locked into JDK 11 for Connect for version 4.0? It would
> > require people to upgrade their connectors to run on JDK 17 but shipping
> > Kafka 4.0 with a Jetty version that is already end of life doesn't make
> > sense to me. I know that Connect supports isolated classloaders for
> > connectors but that of course is not the same as different Java versions.
> >
> > Chris
> >
> > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris
>  > >
> > wrote:
> >
> > > Hi Christopher!
> > >
> > > Thanks so much for raising this. I agree that we should move to the
> > > new namespace in 4.0, and not doing so would be a mistake.
> > > This breaking change has a lot of benefits, and the only cost I am
> > > aware of is that ConnectRestExtensions will need to be migrated,
> > > rebuilt, and re-released for 4.0+
> > >
> > > Can you explain how KIP-1013 and the Java version are relevant?
> > > Connect is dependent on this namespace, but will still need to support
> > > Java 11 in 4.0.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > >  wrote:
> > > >
> > > > Is this already being planned for version 4.0? If not, I strongly
> thing
> > > it
> > > > should be.
> > > >
> > > > Kafka is currently using the old long deprecated javax apis which is
> > > going
> > > > to continue to cause issues [1] for people as more and more things
> are
> > > > updated to use Jakarta.
> > > >
> > > > With the bump to require JDK 17 for version 4.0 [2] this seems like
> the
> > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis
> and
> > > new
> > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that
> > all
> > > > support the new namespace. It needs to be upgraded at some point
> > anyways
> > > so
> > > > a major version makes sense to me.
> > > >
> > > > Another scenario where I've run into this problem is testing. For
> > > example,
> > > > If I try to run tests against my custom code with an embedded Kafka
> > > broker
> > > > and components in JUnit, then things break with newer dependencies
> like
> > > > Spring that require Jakarta as it interferes on the classpath.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > > > [2]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> > >
> >
>

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon]

Greg,

I definitely understand that part, it's always tricky for users that have
older software and haven't or can't upgrade yet.

The most important thing to me is getting to the new Jakarta APIs and Java
EE updates as that causes incompatibilities with code and requires code
changes. So if that requires going with Jetty 11.x for now, that is better
than sticking with 9.4.x at least. If someone wanted to run Jetty 12 I
wouldn't expect it to be a huge deal for someone if they really wanted to
do that and swap it out (especially if using Maven or Gradle to create a
custom build) as it wouldn't require code changes.

I just checked again and it looks like while Jetty did end community
support for 11.x in January, they are at least still providing security
fixes for 2024 [1] and maybe beyond which is the most important thing. It
would be a terrible idea to include something like a web server that
wouldn't receive CVE updates.

I am not sure without diving in what else would need to be upgraded for but
it looks like there's actually an old PR [2] that was closed to migrate to
Jetty 11 and Jakarta (as well as other necessary updates like Jersey) maybe
that could be reopened?

Also, I assume this would need a KIP to proceed.

Chris


[1] https://github.com/jetty/jetty.project/issues/10485
[2] https://github.com/apache/kafka/pull/10176

On Tue, Mar 26, 2024 at 1:11 PM Greg Harris 
wrote:

> Hey Christopher,
>
> Thank you for explaining that jetty version dependency, I was not aware of
> that.
>
> > So are we really locked into JDK 11 for Connect for version 4.0?
>
> Until we adopt a KIP to change the minimum version, we must support 11
> in 4.0. By default we will continue to support Java versions after a
> major version upgrade.
>
> There could be some friction for trying to get such a KIP passed given
> the ecosystem of connectors, their maintenance practices, and how soon
> 4.0 is planned. If it doesn't get adopted, we could also use this
> version dependency as a motivation for having 5.0 sooner rather than
> later, and use the intervening time to properly notify downstream
> users to upgrade their Java versions.
>
> Thanks,
> Greg
>
> On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon
>  wrote:
> >
> > Hi Greg,
> >
> > You are right that KIP-1013 and the JDK 17 upgrade is not directly
> relevant
> > because JDK 11 can still be used with Jakarta APIs. However, it's still
> > somewhat relevant and important because if we are stuck at JDK 11 then we
> > can't upgrade to certain versions. For Connect, there is a Jetty server
> for
> > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> > problem with using Jetty 11.x is that it is already EOL.
> >
> > So are we really locked into JDK 11 for Connect for version 4.0? It would
> > require people to upgrade their connectors to run on JDK 17 but shipping
> > Kafka 4.0 with a Jetty version that is already end of life doesn't make
> > sense to me. I know that Connect supports isolated classloaders for
> > connectors but that of course is not the same as different Java versions.
> >
> > Chris
> >
> > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris
> 
> > wrote:
> >
> > > Hi Christopher!
> > >
> > > Thanks so much for raising this. I agree that we should move to the
> > > new namespace in 4.0, and not doing so would be a mistake.
> > > This breaking change has a lot of benefits, and the only cost I am
> > > aware of is that ConnectRestExtensions will need to be migrated,
> > > rebuilt, and re-released for 4.0+
> > >
> > > Can you explain how KIP-1013 and the Java version are relevant?
> > > Connect is dependent on this namespace, but will still need to support
> > > Java 11 in 4.0.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > >  wrote:
> > > >
> > > > Is this already being planned for version 4.0? If not, I strongly
> thing
> > > it
> > > > should be.
> > > >
> > > > Kafka is currently using the old long deprecated javax apis which is
> > > going
> > > > to continue to cause issues [1] for people as more and more things
> are
> > > > updated to use Jakarta.
> > > >
> > > > With the bump to require JDK 17 for version 4.0 [2] this seems like
> the
> > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis
> and
> > > new
> > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc
> that all
> > > > support the new namespa

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon]

Hi Greg,

You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant
because JDK 11 can still be used with Jakarta APIs. However, it's still
somewhat relevant and important because if we are stuck at JDK 11 then we
can't upgrade to certain versions. For Connect, there is a Jetty server for
the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
problem with using Jetty 11.x is that it is already EOL.

So are we really locked into JDK 11 for Connect for version 4.0? It would
require people to upgrade their connectors to run on JDK 17 but shipping
Kafka 4.0 with a Jetty version that is already end of life doesn't make
sense to me. I know that Connect supports isolated classloaders for
connectors but that of course is not the same as different Java versions.

Chris

On Tue, Mar 26, 2024 at 11:33 AM Greg Harris 
wrote:

> Hi Christopher!
>
> Thanks so much for raising this. I agree that we should move to the
> new namespace in 4.0, and not doing so would be a mistake.
> This breaking change has a lot of benefits, and the only cost I am
> aware of is that ConnectRestExtensions will need to be migrated,
> rebuilt, and re-released for 4.0+
>
> Can you explain how KIP-1013 and the Java version are relevant?
> Connect is dependent on this namespace, but will still need to support
> Java 11 in 4.0.
>
> Thanks!
> Greg
>
> On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
>  wrote:
> >
> > Is this already being planned for version 4.0? If not, I strongly thing
> it
> > should be.
> >
> > Kafka is currently using the old long deprecated javax apis which is
> going
> > to continue to cause issues [1] for people as more and more things are
> > updated to use Jakarta.
> >
> > With the bump to require JDK 17 for version 4.0 [2] this seems like the
> > perfect time to upgrade to a new version of JavaEE and Jakarta apis and
> new
> > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
> > support the new namespace. It needs to be upgraded at some point anyways
> so
> > a major version makes sense to me.
> >
> > Another scenario where I've run into this problem is testing. For
> example,
> > If I try to run tests against my custom code with an embedded Kafka
> broker
> > and components in JUnit, then things break with newer dependencies like
> > Spring that require Jakarta as it interferes on the classpath.
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > [2]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
>

[DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon]

Is this already being planned for version 4.0? If not, I strongly thing it
should be.

Kafka is currently using the old long deprecated javax apis which is going
to continue to cause issues [1] for people as more and more things are
updated to use Jakarta.

With the bump to require JDK 17 for version 4.0 [2] this seems like the
perfect time to upgrade to a new version of JavaEE and Jakarta apis and new
versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
support the new namespace. It needs to be upgraded at some point anyways so
a major version makes sense to me.

Another scenario where I've run into this problem is testing. For example,
If I try to run tests against my custom code with an embedded Kafka broker
and components in JUnit, then things break with newer dependencies like
Spring that require Jakarta as it interferes on the classpath.

[1] https://issues.apache.org/jira/browse/KAFKA-16326
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510

Re: [DISCUSS] Minimum constraint for segment.ms

[Christopher Shannon]

I think it's a great idea to raise a KIP to look at adjusting defaults and
minimum/maximum config values for version 4.0.

As pointed out, the minimum values for segment.ms and segment.bytes don't
make sense and would probably bring down a cluster pretty quickly if set
that low, so version 4.0 is a good time to fix it and to also look at the
other configs as well for adjustments.

On Wed, Mar 13, 2024 at 4:39 AM Sergio Daniel Troiano
 wrote:

> hey guys,
>
> Regarding to num.recovery.threads.per.data.dir: I agree, in our company we
> use the number of vCPUs to do so as this is not competing with ready
> cluster traffic.
>
>
> On Wed, 13 Mar 2024 at 09:29, Luke Chen  wrote:
>
> > Hi Divij,
> >
> > Thanks for raising this.
> > The valid minimum value 1 for `segment.ms` is completely unreasonable.
> > Similarly for `segment.bytes`, `metadata.log.segment.ms`,
> > `metadata.log.segment.bytes`.
> >
> > In addition to that, there are also some config default values we'd like
> to
> > propose to change in v4.0.
> > We can collect more comments from the community, and come out with a KIP
> > for them.
> >
> > 1. num.recovery.threads.per.data.dir:
> > The current default value is 1. But the log recovery is happening before
> > brokers are in ready state, which means, we should use all the available
> > resource to speed up the log recovery to bring the broker to ready state
> > soon. Default value should be... maybe 4 (to be decided)?
> >
> > 2. Other configs might be able to consider to change the default, but
> open
> > for comments:
> >2.1. num.replica.fetchers: default is 1, but that's not enough when
> > there are multiple partitions in the cluster
> >2.2. `socket.send.buffer.bytes`/`socket.receive.buffer.bytes`:
> > Currently, we set 100kb as default value, but that's not enough for
> > high-speed network.
> >
> > Thank you.
> > Luke
> >
> >
> > On Tue, Mar 12, 2024 at 1:32 AM Divij Vaidya 
> > wrote:
> >
> > > Hey folks
> > >
> > > Before I file a KIP to change this in 4.0, I wanted to understand the
> > > historical context for the value of the following setting.
> > >
> > > Currently, segment.ms minimum threshold is set to 1ms [1].
> > >
> > > Segments are expensive. Every segment uses multiple file descriptors
> and
> > > it's easy to run out of OS limits when creating a large number of
> > segments.
> > > Large number of segments also delays log loading on startup because of
> > > expensive operations such as iterating through all directories &
> > > conditionally loading all producer state.
> > >
> > > I am currently not aware of a reason as to why someone might want to
> work
> > > with a segment.ms of less than ~10s (number chosen arbitrary that
> looks
> > > sane)
> > >
> > > What was the historical context of setting the minimum threshold to 1ms
> > for
> > > this setting?
> > >
> > > [1]
> https://kafka.apache.org/documentation.html#topicconfigs_segment.ms
> > >
> > > --
> > > Divij Vaidya
> > >
> >
>

Re: [DISCUSS] Road to Kafka 4.0

[Christopher Shannon]

I think JBOD definitely needs to be before 4.0. That has been a blocker
issue this entire time for me and my team and I'm sure others. While Kraft
has been technically "production ready" for a while, I haven't been able to
upgrade because of missing JBOD support.

On Wed, Oct 11, 2023 at 12:15 PM Ismael Juma  wrote:

> Hi Luke,
>
> This is a good discussion. And there is a lot more to it than KRaft.
>
> With regards to KRaft, there are two separate items:
> 1. Bugs
> 2. Missing features when compared to ZK
>
> When it comes to bugs, I don't see why 4.0 is particularly relevant. KRaft
> has been considered production-ready for over a year. If the bug is truly
> critical, we should fix it for 3.6.1 or 3.7.0 (depending on the
> complexity).
>
> When it comes to missing features, it would be preferable to land them
> before 4.0 as well (ideally 3.7). I believe KIP-858 (JBOD) is the obvious
> one in this category, but there are a few more in your list worth
> discussing.
>
> Ismael
>
> On Wed, Oct 11, 2023 at 5:18 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > While Kafka 3.6.0 is released, I’d like to start the discussion for the
> > “road to Kafka 4.0”. Based on the plan in KIP-833
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.7
> > >,
> > the next release 3.7 will be the final release before moving to Kafka 4.0
> > to remove the Zookeeper from Kafka. Before making this major change, I'd
> > like to get consensus on the "must-have features/fixes for Kafka 4.0", to
> > avoid some users being surprised when upgrading to Kafka 4.0. The intent
> is
> > to have a clear communication about what to expect in the following
> months.
> > In particular we should be signaling what features and configurations are
> > not supported, or at risk (if no one is able to add support or fix known
> > bugs).
> >
> > Here is the JIRA tickets list
> > 
> I
> > labeled for "4.0-blocker". The criteria I labeled as “4.0-blocker” are:
> > 1. The feature is supported in Zookeeper Mode, but not supported in KRaft
> > mode, yet (ex: KIP-858: JBOD in KRaft)
> > 2. Critical bugs in KRaft, (ex: KAFKA-15489 : split brain in KRaft
> > controller quorum)
> >
> > If you disagree with my current list, welcome to have discussion in the
> > specific JIRA ticket. Or, if you think there are some tickets I missed,
> > welcome to start a discussion in the JIRA ticket and ping me or other
> > people. After we get the consensus, we can label/unlabel it afterwards.
> > Again, the goal is to have an open communication with the community about
> > what will be coming in 4.0.
> >
> > Below is the high level category of the list content:
> >
> > 1. Recovery from disk failure
> > KIP-856
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-856:+KRaft+Disk+Failure+Recovery
> > >:
> > KRaft Disk Failure Recovery
> >
> > 2. Prevote to support controllers more than 3
> > KIP-650
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-650%3A+Enhance+Kafkaesque+Raft+semantics
> > >:
> > Enhance Kafkaesque Raft semantics
> >
> > 3. JBOD support
> > KIP-858
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft
> > >:
> > Handle
> > JBOD broker disk failure in KRaft
> >
> > 4. Scale up/down Controllers
> > KIP-853
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > >:
> > KRaft Controller Membership Changes
> >
> > 5. Modifying dynamic configurations on the KRaft controller
> >
> > 6. Critical bugs in KRaft
> >
> > Does this make sense?
> > Any feedback is welcomed.
> >
> > Thank you.
> > Luke
> >
>

Re: [VOTE] 3.2.1 RC3

[Christopher Shannon]

+1 (non binding)

I built from source and ran through some of the tests including all the
connect runtime tests. I verified that KAFKA-14079 was included and the fix
looked good in my tests.

On Thu, Jul 21, 2022 at 9:15 PM David Arthur  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first release candidate of Apache Kafka 3.2.1.
>
> This is a bugfix release with several fixes since the release of 3.2.0. A
> few of the major issues include:
>
> * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck
>
>
> Release notes for the 3.2.1 release:
> https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/RELEASE_NOTES.html
>
>
>
>  Please download, test and vote by Wednesday July 27, 2022 at 17:00 PT.
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/
>
> Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> Javadoc: https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/javadoc/
>
> Tag to be voted upon (off 3.2 branch) is the 3.2.1 tag:
> https://github.com/apache/kafka/releases/tag/3.2.1-rc3
>
> Documentation: https://kafka.apache.org/32/documentation.html
>
> Protocol: https://kafka.apache.org/32/protocol.html
>
>
> The past few builds have had flaky test failures. I will update this thread
> with passing build links soon.
>
> Unit/Integration test job:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/
> System test job:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/
>
>
> Thanks!
> David Arthur
>

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Christopher Shannon]

I submitted a tweaked PR for 3.3.0 based on the 3.2.1 fix here:
https://github.com/apache/kafka/pull/12415

On Sun, Jul 17, 2022 at 1:52 PM Chris Egerton 
wrote:

> Hi all,
>
> Although KAFKA-14079 is not a regression, it'd be nice if we could get it
> into 3.3.0. As I mentioned on the 3.2.1 release thread, the risk is fairly
> low (the functional changes are just two lines long), and the impact is
> high for users who have configured source connectors with
> "errors.tolerance" set to "all".
>
> Given that the code freeze for the release is coming up soon (the release
> doc currently has it at July 20th), it'd be nice if we could get some eyes
> on the PR for 3.2.1 and, if that looks good, a PR for trunk and 3.3.0.
>
> Cheers,
>
> Chris
>
> On Sat, Jul 16, 2022 at 12:31 PM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > There is a bug I found that I think is worthwhile fixing in 3.3.0 (I also
> > sent a note to 3.2.1 thread):
> > https://issues.apache.org/jira/browse/KAFKA-14079
> >
> > On Thu, Jul 14, 2022 at 7:56 PM Jason Gustafson
>  > >
> > wrote:
> >
> > > Hey Jose,
> > >
> > > Thanks for volunteering to manage the release! KIP-833 is currently
> > slotted
> > > for 3.3. We've been getting some help from Jack Vanlighty to validate
> the
> > > raft implementation in TLA+ and with frameworks like Jepsen. The
> > > specification is written here if anyone is interested:
> > >
> > >
> >
> https://github.com/Vanlightly/raft-tlaplus/blob/main/specifications/pull-raft/KRaft.tla
> > > .
> > > The main gap that this work uncovered in our implementation is
> documented
> > > here: https://issues.apache.org/jira/browse/KAFKA-14077. I do believe
> > that
> > > KIP-833 depends on fixing this issue, so I wanted to see how you feel
> > about
> > > giving us a little more time to address it?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Jul 13, 2022 at 10:01 AM Sagar 
> > wrote:
> > >
> > > > Hey Jose,
> > > >
> > > > Well actually I have 2 approved PRs from Kafka Connect:
> > > >
> > > > https://github.com/apache/kafka/pull/12321
> > > > https://github.com/apache/kafka/pull/12309
> > > >
> > > > Not sure how to get these merged though but I think these can go into
> > 3.3
> > > > release.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > >
> > > > On Wed, Jul 13, 2022 at 5:03 PM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Jose
> > > > >
> > > > > A few of my PRs are pending review for quite some which I was
> hoping
> > to
> > > > > merge into 3.3. I have already marked them with "Fix version=3.3.0"
> > so
> > > > that
> > > > > you can track them using the JIRA filter you shared earlier
> > > > > <
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.3.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20%20%20%20%20
> > > > > >
> > > > > in this thread. Would you have some time to review them?
> > > > >
> > > > > Notable amongst them would be:
> > > > > 1. Fix the rate window size calculation for edge cases -
> > > > > https://github.com/apache/kafka/pull/12184
> > > > > 2. Fix resource leaks - https://github.com/apache/kafka/pull/12228
> > > > >
> > > > > And the complete list would be at:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/search?q=is%3Aopen+is%3Apr+author%3Adivijvaidya+is%3Apr+repo%3Aapache%2Fkafka+created%3A2022-04-01..2022-07-30=Issues
> > > > >
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jul 11, 2022 at 5:12 PM José Armando García Sancio
> > > > >  wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I created the branch for 3.3
> > > > > > (https://github.com/apache/kafka/tree/3.3). If you have bug
> fixes
> > > for
> > > > > > the 3.3.0 release please make sure to cherry pick them to that
> > > branch.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.2.1 release

[Christopher Shannon]

Hi Chris E.

Thanks for all the feedback earlier, I updated the PR based on your
comments and also pushed a second PR for trunk for 3.3.0

I agree the impact is high which is why I found the issue. I recently
turned on this feature and suddenly my connect worker/task kept
periodically falling over with OOM errors. Finally I took a heap dump and
saw a ton of submitted record objects in memory and started investigating
and that's how I figured out it was related to this new feature..

Chris

On Sun, Jul 17, 2022 at 1:48 PM Chris Egerton 
wrote:

> Hi Chris,
>
> Good find, and thanks for filing a fix. I agree that we should get this
> into 3.2.1 if possible. The risk is fairly low (the functional parts of the
> fix are just two lines long) and the impact of the bug is high for users
> who have configured source connectors with "errors.tolerance" set to "all".
>
> Cheers,
>
> Chris
>
> On Sat, Jul 16, 2022 at 12:26 PM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > HI,
> >
> > I think I found a memory leak that was introduced in 3.2.0 in a Connector
> > SourceTask. I created a JIRA:
> > https://issues.apache.org/jira/browse/KAFKA-14079 and small PR with a
> fix:
> > https://github.com/apache/kafka/pull/12412
> >
> > I think this should be included in 3.2.1. It should also go into 3.3.0
> but
> > there was a lot of refactoring done there with the source task code due
> to
> > KIP-618 so another PR needs to be done for that if this is merged.
> >
> > Chris
> >
> > On Fri, Jul 15, 2022 at 10:06 AM David Arthur  wrote:
> >
> > > Here is the release plan for 3.2.1
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.1
> > >
> > > I am working on getting clarity on the one open blocker. Once that is
> > > resolved (or rescheduled for a future release), I will build the first
> > > release candidate.
> > >
> > > -David
> > >
> > > On Thu, Jul 14, 2022 at 3:10 AM Luke Chen  wrote:
> > >
> > > > +1, Thanks David!
> > > >
> > > > On Thu, Jul 14, 2022 at 1:16 PM David Jacot 
> > > wrote:
> > > >
> > > > > +1. Thanks David.
> > > > >
> > > > > Le mer. 13 juil. 2022 à 23:43, José Armando García Sancio
> > > > >  a écrit :
> > > > >
> > > > > > +1. Thanks for volunteering David.
> > > > > >
> > > > > > --
> > > > > > -José
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Christopher Shannon]

There is a bug I found that I think is worthwhile fixing in 3.3.0 (I also
sent a note to 3.2.1 thread):
https://issues.apache.org/jira/browse/KAFKA-14079

On Thu, Jul 14, 2022 at 7:56 PM Jason Gustafson 
wrote:

> Hey Jose,
>
> Thanks for volunteering to manage the release! KIP-833 is currently slotted
> for 3.3. We've been getting some help from Jack Vanlighty to validate the
> raft implementation in TLA+ and with frameworks like Jepsen. The
> specification is written here if anyone is interested:
>
> https://github.com/Vanlightly/raft-tlaplus/blob/main/specifications/pull-raft/KRaft.tla
> .
> The main gap that this work uncovered in our implementation is documented
> here: https://issues.apache.org/jira/browse/KAFKA-14077. I do believe that
> KIP-833 depends on fixing this issue, so I wanted to see how you feel about
> giving us a little more time to address it?
>
> Thanks,
> Jason
>
> On Wed, Jul 13, 2022 at 10:01 AM Sagar  wrote:
>
> > Hey Jose,
> >
> > Well actually I have 2 approved PRs from Kafka Connect:
> >
> > https://github.com/apache/kafka/pull/12321
> > https://github.com/apache/kafka/pull/12309
> >
> > Not sure how to get these merged though but I think these can go into 3.3
> > release.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Wed, Jul 13, 2022 at 5:03 PM Divij Vaidya 
> > wrote:
> >
> > > Hey Jose
> > >
> > > A few of my PRs are pending review for quite some which I was hoping to
> > > merge into 3.3. I have already marked them with "Fix version=3.3.0" so
> > that
> > > you can track them using the JIRA filter you shared earlier
> > > <
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.3.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20%20%20%20%20
> > > >
> > > in this thread. Would you have some time to review them?
> > >
> > > Notable amongst them would be:
> > > 1. Fix the rate window size calculation for edge cases -
> > > https://github.com/apache/kafka/pull/12184
> > > 2. Fix resource leaks - https://github.com/apache/kafka/pull/12228
> > >
> > > And the complete list would be at:
> > >
> > >
> >
> https://github.com/search?q=is%3Aopen+is%3Apr+author%3Adivijvaidya+is%3Apr+repo%3Aapache%2Fkafka+created%3A2022-04-01..2022-07-30=Issues
> > >
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Mon, Jul 11, 2022 at 5:12 PM José Armando García Sancio
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I created the branch for 3.3
> > > > (https://github.com/apache/kafka/tree/3.3). If you have bug fixes
> for
> > > > the 3.3.0 release please make sure to cherry pick them to that
> branch.
> > > >
> > > > Thanks
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.2.1 release

[Christopher Shannon]

HI,

I think I found a memory leak that was introduced in 3.2.0 in a Connector
SourceTask. I created a JIRA:
https://issues.apache.org/jira/browse/KAFKA-14079 and small PR with a fix:
https://github.com/apache/kafka/pull/12412

I think this should be included in 3.2.1. It should also go into 3.3.0 but
there was a lot of refactoring done there with the source task code due to
KIP-618 so another PR needs to be done for that if this is merged.

Chris

On Fri, Jul 15, 2022 at 10:06 AM David Arthur  wrote:

> Here is the release plan for 3.2.1
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.1
>
> I am working on getting clarity on the one open blocker. Once that is
> resolved (or rescheduled for a future release), I will build the first
> release candidate.
>
> -David
>
> On Thu, Jul 14, 2022 at 3:10 AM Luke Chen  wrote:
>
> > +1, Thanks David!
> >
> > On Thu, Jul 14, 2022 at 1:16 PM David Jacot 
> wrote:
> >
> > > +1. Thanks David.
> > >
> > > Le mer. 13 juil. 2022 à 23:43, José Armando García Sancio
> > >  a écrit :
> > >
> > > > +1. Thanks for volunteering David.
> > > >
> > > > --
> > > > -José
> > > >
> > >
> >
>
>
> --
> David Arthur
>

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

[Christopher Shannon]

Revisiting this as this has come up for my use case again. Specifically for
validation I need to be able to validate headers including compressed
messages. It looks like in LogValidator the messages are already
decompressed to validate records but the headers get skipped when loaded
into a partial record. So as part of this change I would think there should
be a way to read in the headers for validation even if records are
compressed.

On Wed, Jul 7, 2021 at 3:30 AM Nikolay Izhikov  wrote:

> Hello, James.
>
> > One use case we would like is to require that producers are sending
> compressed messages.
>
> I think that forcing producers to send compressed messages is out of scope
> of this KIP.
>
>
> > 7 июля 2021 г., в 08:48, Soumyajit Sahu 
> написал(а):
> >
> > Interesting point. You are correct that at least KIP-729 cannot validate
> > that.
> >
> > We could propose a different KIP for that which could enforce that in the
> > upper layer. Personally, I would be hesitant to discard the data in that
> > case, but just use metrics/logs to detect those and inform the producers
> > about it.
> >
> >
> > On Tue, Jul 6, 2021, 9:13 PM James Cheng  wrote:
> >
> >> One use case we would like is to require that producers are sending
> >> compressed messages. Would this KIP (or KIP-686) allow the broker to
> detect
> >> that? From looking at both KIPs, it doesn't look it would help with my
> >> particular use case. Both of the KIPs are at the Record-level.
> >>
> >> Thanks,
> >> -James
> >>
> >>> On Jun 30, 2021, at 10:05 AM, Soumyajit Sahu  >
> >> wrote:
> >>>
> >>> Hi Nikolay,
> >>> Great to hear that. I'm ok with either one too.
> >>> I had missed noticing the KIP-686. Thanks for bringing it up.
> >>>
> >>> I have tried to keep this one simple, but hope it can cover all our
> >>> enterprise needs.
> >>>
> >>> Should we put this one for vote?
> >>>
> >>> Regards,
> >>> Soumyajit
> >>>
> >>>
> >>> On Wed, Jun 30, 2021, 8:50 AM Nikolay Izhikov 
> >> wrote:
> >>>
> >>>> Team, If we have support from committers for API to check records on
> the
> >>>> broker side let’s choose one KIP to go with and move forward to vote
> and
> >>>> implementation?
> >>>> I’m ready to drive implementation of this API.
> >>>>
> >>>> I’m ready to drive the implementation of this API.
> >>>> It seems very useful to me.
> >>>>
> >>>>> 30 июня 2021 г., в 18:04, Nikolay Izhikov 
> >>>> написал(а):
> >>>>>
> >>>>> Hello.
> >>>>>
> >>>>> I had a very similar proposal [1].
> >>>>> So, yes, I think we should have one implementation of API in the
> >> product.
> >>>>>
> >>>>> [1]
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-686%3A+API+to+ensure+Records+policy+on+the+broker
> >>>>>
> >>>>>> 30 июня 2021 г., в 17:57, Christopher Shannon <
> >>>> christopher.l.shan...@gmail.com> написал(а):
> >>>>>>
> >>>>>> I would find this feature very useful as well as adding custom
> >>>> validation
> >>>>>> to incoming records would be nice to prevent bad data from making it
> >> to
> >>>> the
> >>>>>> topic.
> >>>>>>
> >>>>>> On Wed, Apr 7, 2021 at 7:03 PM Soumyajit Sahu <
> >> soumyajit.s...@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Colin! Good call on the ApiRecordError. We could use
> >>>>>>> InvalidRecordException instead, and have the broker convert it
> >>>>>>> to ApiRecordError.
> >>>>>>> Modified signature below.
> >>>>>>>
> >>>>>>> interface BrokerRecordValidator {
> >>>>>>> /**
> >>>>>>> * Validate the record for a given topic-partition.
> >>>>>>> */
> >>>>>>> Optional validateRecord(TopicPartition
> >>>>>>> topicPartition, ByteBuffer key, ByteBuffer value, Header[]
> headers);
> >>>>>>> }
> >>>

Re: [VOTE] KIP-779: Allow Source Tasks to Handle Producer Exceptions

[Christopher Shannon]

+1 (non-binding). This looks good to me and will be useful as a way to
handle producer errors.

On Mon, Nov 8, 2021 at 8:55 AM Knowles Atchison Jr 
wrote:

> Good morning,
>
> I'd like to start a vote for KIP-779: Allow Source Tasks to Handle Producer
> Exceptions:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-779%3A+Allow+Source+Tasks+to+Handle+Producer+Exceptions
>
> The purpose of this KIP is to allow Source Tasks the option to "ignore"
> kafka producer exceptions. After a few iterations, this is now part of the
> "errors.tolerance" configuration and provides a null RecordMetadata to
> commitRecord() in lieu of a new SourceTask interface method or worker
> configuration item.
>
> PR is here:
>
> https://github.com/apache/kafka/pull/11382
>
> Any comments and feedback are welcome.
>
> Knowles
>

Re: [DISCUSS] KIP-779: Allow Source Tasks to Handle Producer Exceptions

[Christopher Shannon]

I also would find this feature useful to handle errors better, does anyone
have any comments or feedback?


On Mon, Oct 11, 2021 at 8:52 AM Knowles Atchison Jr 
wrote:

> Good morning,
>
> Bumping this for visibility. I would like this to go into the next release.
> KIP freeze is Friday.
>
> Any comments and feedback are welcome.
>
> Knowles
>
> On Tue, Oct 5, 2021 at 4:24 PM Knowles Atchison Jr 
> wrote:
>
> > Hello all,
> >
> > I would like to discuss the following KIP:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-779%3A+Allow+Source+Tasks+to+Handle+Producer+Exceptions
> >
> > The main purpose is to allow Source Tasks the ability to see underlying
> > Producer Exceptions and decide what to do rather than being killed. In
> our
> > use cases we would want to log/write off some information and continue
> > processing.
> >
> > PR is here:
> >
> > https://github.com/apache/kafka/pull/11382
> >
> > Any comments and feedback are welcome.
> >
> >
> > Knowles
> >
>

Re: Ensuring that Kafka message is persisted after acknowledgement

[Christopher Shannon]

You can control fsync with two properties:

https://kafka.apache.org/documentation/#topicconfigs_flush.messages
https://kafka.apache.org/documentation/#topicconfigs_flush.ms


On Wed, Aug 25, 2021 at 10:39 AM Kunal Goyal 
wrote:

> Hello,
>
> We are exploring using Kafka for our application. Our requirement is that
> once we write some messages to Kafka, it should be guaranteed that the
> messages are persisted to disk.
> We found this
> <
> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
> >
> article
> which says that a Kafka broker acknowledges a record after it has written
> the record to the buffer of the I/O device; it does not issue an explicit
> fsync operation nor does it wait for the OS to confirm that the data has
> been written. Is this statement true for the current implementation? If so,
> is there any way in which we can ensure fsync is called before
> acknowledgement of messages?
> Any help would be appreciated.
>
> --
>
> Thanks & Regards
>
> Kunal Goyal
>

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

[Christopher Shannon]

I would find this feature very useful as well as adding custom validation
to incoming records would be nice to prevent bad data from making it to the
topic.

On Wed, Apr 7, 2021 at 7:03 PM Soumyajit Sahu 
wrote:

> Thanks Colin! Good call on the ApiRecordError. We could use
> InvalidRecordException instead, and have the broker convert it
> to ApiRecordError.
> Modified signature below.
>
> interface BrokerRecordValidator {
>/**
> * Validate the record for a given topic-partition.
> */
> Optional validateRecord(TopicPartition
> topicPartition, ByteBuffer key, ByteBuffer value, Header[] headers);
> }
>
> On Tue, Apr 6, 2021 at 5:09 PM Colin McCabe  wrote:
>
> > Hi Soumyajit,
> >
> > The difficult thing is deciding which fields to share and how to share
> > them.  Key and value are probably the minimum we need to make this
> useful.
> > If we do choose to go with byte buffer, it is not necessary to also pass
> > the size, since ByteBuffer maintains that internally.
> >
> > ApiRecordError is also an internal class, so it can't be used in a public
> > API.  I think most likely if we were going to do this, we would just
> catch
> > an exception and use the exception text as the validation error.
> >
> > best,
> > Colin
> >
> >
> > On Tue, Apr 6, 2021, at 15:57, Soumyajit Sahu wrote:
> > > Hi Tom,
> > >
> > > Makes sense. Thanks for the explanation. I get what Colin had meant
> > earlier.
> > >
> > > Would a different signature for the interface work? Example below, but
> > > please feel free to suggest alternatives if there are any possibilities
> > of
> > > such.
> > >
> > > If needed, then deprecating this and introducing a new signature would
> be
> > > straight-forward as both (old and new) calls could be made serially in
> > the
> > > LogValidator allowing a coexistence for a transition period.
> > >
> > > interface BrokerRecordValidator {
> > > /**
> > >  * Validate the record for a given topic-partition.
> > >  */
> > > Optional validateRecord(TopicPartition
> > topicPartition,
> > > int keySize, ByteBuffer key, int valueSize, ByteBuffer value, Header[]
> > > headers);
> > > }
> > >
> > >
> > > On Tue, Apr 6, 2021 at 12:54 AM Tom Bentley 
> wrote:
> > >
> > > > Hi Soumyajit,
> > > >
> > > > Although that class does indeed have public access at the Java level,
> > it
> > > > does so only because it needs to be used by internal Kafka code which
> > lives
> > > > in other packages (there isn't any more restrictive access modifier
> > which
> > > > would work). What the project considers public Java API is determined
> > by
> > > > what's included in the published Javadocs:
> > > > https://kafka.apache.org/27/javadoc/index.html, which doesn't
> include
> > the
> > > > org.apache.kafka.common.record package.
> > > >
> > > > One of the problems with making these internal classes public is it
> > ties
> > > > the project into supporting them as APIs, which can make changing
> them
> > much
> > > > harder and in the long run that can slow, or even prevent, innovation
> > in
> > > > the rest of Kafka.
> > > >
> > > > Kind regards,
> > > >
> > > > Tom
> > > >
> > > >
> > > >
> > > > On Sun, Apr 4, 2021 at 7:31 PM Soumyajit Sahu <
> > soumyajit.s...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Colin,
> > > > > I see that both the interface "Record" and the implementation
> > > > > "DefaultRecord" being used in LogValidator.java are public
> > > > > interfaces/classes.
> > > > >
> > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/Records.java
> > > > > and
> > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/DefaultRecord.java
> > > > >
> > > > > So, it should be ok to use them. Let me know what you think.
> > > > >
> > > > > Thanks,
> > > > > Soumyajit
> > > > >
> > > > >
> > > > > On Fri, Apr 2, 2021 at 8:51 AM Colin McCabe 
> > wrote:
> > > > >
> > > > > > Hi Soumyajit,
> > > > > >
> > > > > > I believe we've had discussions about proposals similar to this
> > before,
> > > > > > although I'm having trouble finding one right now.  The issue
> here
> > is
> > > > > that
> > > > > > Record is a private class -- it is not part of any public API,
> and
> > may
> > > > > > change at any time.  So we can't expose it in public APIs.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 1, 2021, at 14:18, Soumyajit Sahu wrote:
> > > > > > > Hello All,
> > > > > > > I would like to start a discussion on the KIP-729.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-729%3A+Custom+validation+of+records+on+the+broker+prior+to+log+append
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Soumyajit
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-755: Add new AUTO_CREATE ACL for auto topic creation

[Christopher Shannon]

As a quick follow up after I sent my last response I realized that passing
the principal to the create topic policy still probably doesn't work
because while you have the user you wouldn't know the permissions allowed.
You could mark the principal as a super user or something but the ACL
authorizer is what loads everything so you really need to have access to
the ACLs loaded to control and make decisions per user so I don't know that
passing the principal to the create topic policy really solves the issue.
About the only thing you could do with the principal is just check if it's
a super user or not which is better but not perfect.

On Wed, Jun 23, 2021 at 8:49 AM Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Hi Ismael,
>
> Thanks for the response.
>
> I initially tried to use a create topic policy but the issue with the
> CreateTopicPolicy and AlterConfigPolicy API is the authenticated user
> making the request is not known as it is not passed as part of the metadata
> to the validation request. In my situation I want to allow administrators
> (or other users granted permission) the ability to set explicit topic
> configs but certain other users I would want to deny this ability. So
> essentially I need the following:
>
> 1) Allow admins or super users the ability to create topics and also set
> explicit configs on the topics
> 2) Allow some users the ability to auto-create topics but not set explicit
> configs (only given cluster defaults)
> 3) Deny creation of topics entirely to other users
>
> One alternative could be to pass the user principal to a create topic
> policy and alter config policy. I actually had thought of this but figured
> ACLs were better since the goal here is really more permissions based so
> ACLs seemed like a better fit.
>
> I updated my KIP to add the goals I listed above and also show the option
> to pass a user principal to rejected alternatives. However, if you think it
> is better I am open to going with the different approach of providing the
> user principal as part of the metadata to the CreateTopicPolicy and
> AlterConfigPolicies.
>
> Chris
>
> On Wed, Jun 23, 2021 at 8:27 AM Ismael Juma  wrote:
>
>> Hi Christopher,
>>
>> Thanks for the KIP. Have you considered using create topic policies to
>> enforce the configuration you desire even when create topics is used
>> directly? You could disallow all create topics requests that set explicit
>> topic configs (i.e. don't rely on cluster defaults).
>>
>> This would work better since may rely on the create topics protocol api
>> since it's the recommended way to create topics.
>>
>> Ismael
>>
>> On Tue, Jun 8, 2021, 11:09 AM Christopher Shannon <
>> christopher.l.shan...@gmail.com> wrote:
>>
>> > Hi Everyone,
>> >
>> > I'd like to propose KIP-755: Add new AUTO_CREATE ACL for auto topic
>> > creation
>> >
>> > The KIP can be found here:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-755%3A+Add+new+AUTO_CREATE+ACL+for+auto+topic+creation
>> >
>> > I submitted the following PR:
>> https://github.com/apache/kafka/pull/10845
>> >
>> > I know it might be too late based on the schedule but it's a small
>> change
>> > so it would be nice to get this in time for 3.0.0.
>> >
>> > Thanks,
>> > Chris
>> >
>>
>

Re: [DISCUSS] KIP-755: Add new AUTO_CREATE ACL for auto topic creation

[Christopher Shannon]

Hi Ismael,

Thanks for the response.

I initially tried to use a create topic policy but the issue with the
CreateTopicPolicy and AlterConfigPolicy API is the authenticated user
making the request is not known as it is not passed as part of the metadata
to the validation request. In my situation I want to allow administrators
(or other users granted permission) the ability to set explicit topic
configs but certain other users I would want to deny this ability. So
essentially I need the following:

1) Allow admins or super users the ability to create topics and also set
explicit configs on the topics
2) Allow some users the ability to auto-create topics but not set explicit
configs (only given cluster defaults)
3) Deny creation of topics entirely to other users

One alternative could be to pass the user principal to a create topic
policy and alter config policy. I actually had thought of this but figured
ACLs were better since the goal here is really more permissions based so
ACLs seemed like a better fit.

I updated my KIP to add the goals I listed above and also show the option
to pass a user principal to rejected alternatives. However, if you think it
is better I am open to going with the different approach of providing the
user principal as part of the metadata to the CreateTopicPolicy and
AlterConfigPolicies.

Chris

On Wed, Jun 23, 2021 at 8:27 AM Ismael Juma  wrote:

> Hi Christopher,
>
> Thanks for the KIP. Have you considered using create topic policies to
> enforce the configuration you desire even when create topics is used
> directly? You could disallow all create topics requests that set explicit
> topic configs (i.e. don't rely on cluster defaults).
>
> This would work better since may rely on the create topics protocol api
> since it's the recommended way to create topics.
>
> Ismael
>
> On Tue, Jun 8, 2021, 11:09 AM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > I'd like to propose KIP-755: Add new AUTO_CREATE ACL for auto topic
> > creation
> >
> > The KIP can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-755%3A+Add+new+AUTO_CREATE+ACL+for+auto+topic+creation
> >
> > I submitted the following PR: https://github.com/apache/kafka/pull/10845
> >
> > I know it might be too late based on the schedule but it's a small change
> > so it would be nice to get this in time for 3.0.0.
> >
> > Thanks,
> > Chris
> >
>

Re: [DISCUSS] KIP-755: Add new AUTO_CREATE ACL for auto topic creation

[Christopher Shannon]

Hi Everyone,

It has been a couple weeks so figured I'd ping one last time to see if
anyone had any interest in this. This feature would be quite useful to me
(and hopefully others) and should be a pretty small change and backwards
compatible.

Chris

On Fri, Jun 11, 2021 at 9:24 AM Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Does anyone have any thoughts or feedback on this KIP?
>
> On Tue, Jun 8, 2021 at 2:09 PM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'd like to propose KIP-755: Add new AUTO_CREATE ACL for auto topic
>> creation
>>
>> The KIP can be found here:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-755%3A+Add+new+AUTO_CREATE+ACL+for+auto+topic+creation
>>
>> I submitted the following PR: https://github.com/apache/kafka/pull/10845
>>
>> I know it might be too late based on the schedule but it's a small change
>> so it would be nice to get this in time for 3.0.0.
>>
>> Thanks,
>> Chris
>>
>

Re: [DISCUSS] KIP-755: Add new AUTO_CREATE ACL for auto topic creation

[Christopher Shannon]

Does anyone have any thoughts or feedback on this KIP?

On Tue, Jun 8, 2021 at 2:09 PM Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Hi Everyone,
>
> I'd like to propose KIP-755: Add new AUTO_CREATE ACL for auto topic
> creation
>
> The KIP can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-755%3A+Add+new+AUTO_CREATE+ACL+for+auto+topic+creation
>
> I submitted the following PR: https://github.com/apache/kafka/pull/10845
>
> I know it might be too late based on the schedule but it's a small change
> so it would be nice to get this in time for 3.0.0.
>
> Thanks,
> Chris
>

[DISCUSS] KIP-755: Add new AUTO_CREATE ACL for auto topic creation

[Christopher Shannon]

Hi Everyone,

I'd like to propose KIP-755: Add new AUTO_CREATE ACL for auto topic creation

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-755%3A+Add+new+AUTO_CREATE+ACL+for+auto+topic+creation

I submitted the following PR: https://github.com/apache/kafka/pull/10845

I know it might be too late based on the schedule but it's a small change
so it would be nice to get this in time for 3.0.0.

Thanks,
Chris

Re: Kafka Advisory Topic

[Christopher Shannon]

After discussing this a bit more and looking into what it would take to
implement this feature Knowles and I are going to hold off until we see
what the removal of Zookeeper will look like. We are still interested in
this feature but we figure that things will change significantly enough
that it's probably a good idea to wait to see how things change with Kafka
3.0.

On Tue, Jan 26, 2021 at 1:21 PM Knowles Atchison Jr 
wrote:

> Gwen,
>
> Yes, Chris and I work together; we can put together a KIP.
>
> If someone would please grant me wiki permissions, username is katchison.
>
> Thank you.
>
> On Mon, Jan 25, 2021 at 2:58 PM Gwen Shapira  wrote:
>
> > Agree that this sounds like a good idea.
> >
> > Would be good to have a more formal proposal (i.e. a KIP) with the
> details.
> > I can think of about 100 different questions (will there be "levels"
> > like in logs, what type of events are in or out of scope, rate
> > limiting, data formats, etc).
> > I am also curious on whether the notifications are intended for
> > humans, automated processes or even the Kafka client applications
> > themselves. I hope the proposal can include a few example scenarios to
> > help us reason about the experience.
> >
> > Knowlton, is this something you want to pick up?
> >
> > Gwen
> >
> > On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
> >  wrote:
> > >
> > > Hi,
> > >
> > > I am on the ActiveMQ PMC and I think this is a very good idea to have a
> > way
> > > to do advisories/notifications/events (whatever you want to call it).
> In
> > > ActiveMQ classic you have advisories and in Artemis you have
> > notifications.
> > > Having management messages that can be subscribed to in real time is
> > > actually a major feature that is missing from Kafka that many other
> > brokers
> > > have.
> > >
> > > The idea here would be to publish notifications of different
> configurable
> > > events when something important happens so a consumer can listen in on
> > > things it cares about and be able to do something instead of having to
> > poll
> > > the admin API. There are many events that happen in a broker that would
> > be
> > > useful to be notified about. Events such as new connections to the
> > cluster,
> > > new topics created or destroyed, consumer group creation, authorization
> > > errors, new leader election, etc. The list is pretty much endless.
> > >
> > > The metadata topic that will exist is probably not going to have all of
> > > this information so some other mechanism would be needed to handle
> > > publishing these messages to a specific management topic that would be
> > > useful for a consumer.
> > >
> > > Chris
> > >
> > >
> > > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Hey Knowles,
> > > >
> > > > in Kafka people normally use admin clients to get those metadata. I'm
> > not
> > > > sure why you mentioned specifically that having a topic to manage
> these
> > > > information is useful, but a good news is that in KIP-500
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > > >
> > > > we
> > > > are trying to deprecate Zookeeper and migrate to a self-managed
> > metadata
> > > > topic quorum. At the time this feature is fully done, you should be
> > able to
> > > > use consumers to pull the metadata log.
> > > >
> > > > Best,
> > > > Boyang
> > > >
> > > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > > > katchiso...@gmail.com>
> > > > wrote:
> > > >
> > > > > Good afternoon all,
> > > > >
> > > > > In our Kafka clusters we have a need to know when certain
> activities
> > are
> > > > > performed, mainly topics being created, but brokers coming up/down
> is
> > > > also
> > > > > useful. This would be akin to what ActiveMQ does via advisory
> > messages (
> > > > > https://activemq.apache.org/advisory-message).
> > > > >
> > > > > Since there did not appear to be anything in the ecosystem
> > currently, I
> > > > > wrot

Re: Kafka Advisory Topic

[Christopher Shannon]

Hi,

I am on the ActiveMQ PMC and I think this is a very good idea to have a way
to do advisories/notifications/events (whatever you want to call it). In
ActiveMQ classic you have advisories and in Artemis you have notifications.
Having management messages that can be subscribed to in real time is
actually a major feature that is missing from Kafka that many other brokers
have.

The idea here would be to publish notifications of different configurable
events when something important happens so a consumer can listen in on
things it cares about and be able to do something instead of having to poll
the admin API. There are many events that happen in a broker that would be
useful to be notified about. Events such as new connections to the cluster,
new topics created or destroyed, consumer group creation, authorization
errors, new leader election, etc. The list is pretty much endless.

The metadata topic that will exist is probably not going to have all of
this information so some other mechanism would be needed to handle
publishing these messages to a specific management topic that would be
useful for a consumer.

Chris


On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen 
wrote:

> Hey Knowles,
>
> in Kafka people normally use admin clients to get those metadata. I'm not
> sure why you mentioned specifically that having a topic to manage these
> information is useful, but a good news is that in KIP-500
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >
> we
> are trying to deprecate Zookeeper and migrate to a self-managed metadata
> topic quorum. At the time this feature is fully done, you should be able to
> use consumers to pull the metadata log.
>
> Best,
> Boyang
>
> On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> katchiso...@gmail.com>
> wrote:
>
> > Good afternoon all,
> >
> > In our Kafka clusters we have a need to know when certain activities are
> > performed, mainly topics being created, but brokers coming up/down is
> also
> > useful. This would be akin to what ActiveMQ does via advisory messages (
> > https://activemq.apache.org/advisory-message).
> >
> > Since there did not appear to be anything in the ecosystem currently, I
> > wrote a standalone Java program that watches the various ZooKeeper
> > locations that the Kafka broker writes to and deltas can tell us
> > topic/broker actions etc... and writes to a kafka topic for downstream
> > consumption.
> >
> > Ideally, we would rather have the broker handle this internally rather
> > than yet another service stood up in our systems. I began digging through
> > the broker source (my Scala is basically hello world level) and there
> does
> > not appear to be any mechanism in which this could be easily patched into
> > the broker.
> >
> > Specifically, a producer or consumer acting upon an nonexistent topic or
> a
> > manual CreateTopic would trigger a Produce to this advisory topic and the
> > KafkaApis framework would handle it like any other request. However, by
> the
> > time we are inside the getTopicMetadata call there doesn't seem to be a
> > clean way to fire off another message that would make its way through
> > KafkaApis. Perhaps another XManager type object is required?
> >
> > Looking for alternative ideas or guidance (or I missed something in the
> > broker).
> >
> > Thank you.
> >
> > Knowles
> >
>

Re: [DISCUSS] URIs on Producer and Consumer

[Christopher Shannon]

I think this would be useful as a secondary way to configure.  If others
agree then you can write up a KIP and it can be discussed in more detail.

On Tue, Oct 3, 2017 at 8:56 AM, Clebert Suconic 
wrote:

> Maybe I didn't make the message clear enough...
>
> Would using an URI to the constructor (in addition to the properties)
> help the API, or anyone see a reason to not do it?
>
> KafkaConsumer consumer = new
> KafkaConsumer<>("tcp://localhost:?receive.buffer.bytes=-2", new
> ByteArrayDeserializer(), new ByteArrayDeserializer());
>
> I could send a Pull Request for that. The framework I would write
> would validate if the parameters are valid or not.
>
>
> Thanks in advance
>
>
> On Mon, Oct 2, 2017 at 9:14 AM, Clebert Suconic
>  wrote:
> > At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
> > interesting feature where you can pass parameters through an URI.
> >
> > I was looking at Producer and Consumer APIs, and these two classes are
> > using a method that I considered old for Artemis resembling HornetQ:
> >
> > Instead of passing a Properties (aka HashMaps), users would be able to
> > create a Consumer or Producer by simply doing:
> >
> > new Consumer("tcp::/host:port?properties=values;properties=
> values...etc");
> >
> > Example:
> >
> >
> > Instead of the following:
> >
> > Map config = new HashMap<>();
> > config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:");
> > config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
> > new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
> > ByteArrayDeserializer());
> >
> >
> >
> > Someone could do
> >
> > new KafkaConsumer<>("tcp://localhost:?receive.buffer.bytes=-2",
> > new ByteArrayDeserializer(), new ByteArrayDeserializer());
> >
> >
> >
> > I don't know if that little API improvement would be welcomed? I would be
> > able to send a Pull Request but I don't want to do it if that wouldn't
> > be welcomed in the first place:
> >
> >
> > Just an idea...  let me know if that is welcomed or not.
> >
> > If so I can forward the discussion into how I would implement it.
>
>
>
> --
> Clebert Suconic
>

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Ismael,

I took a closer look at SslTransportLayer and did verify everything will
work for me. SSLSession is already exposed through a public method on
SslTransportLayer through the sslSession() method so that is perfect as I
can access what I need with that.  Mostly I just need the X509 chain out of
the SSLSession for my purposes.  So I went ahead and closed out the JIRA
and moved my KIP to the discarded section now that I verified it will work.

Thanks again for the help.

Chris

On Fri, Feb 24, 2017 at 1:54 PM, Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Sounds good, thanks for the feedback. I will move my KIP to the discarded
> section for now as the PrincipalBuilder should be sufficient.  If for some
> reason it is not I can revisit this.
>
> Chris
>
> On Fri, Feb 24, 2017 at 1:37 PM Ismael Juma <ism...@juma.me.uk> wrote:
>
>> Hi Christopher,
>>
>> It is possible to retrieve the certificates already. The Principal
>> returned
>> by TransportLayer.peerPrincipal() is a X500Principal if TLS is being used.
>> We could add a method to make things nicer, potentially, but just wanted
>> you to know that it's possible today.
>>
>> I am in favour of keeping it simple, if possible. :)
>>
>> As Rajini said, KIP-103 means users have a way to set the config on a per
>> listener basis, so it makes sense to allow users to have both SSL client
>> authentication and SASL authentication enabled at the same time if they
>> wish. As long as the default remains that SSL client auth is disabled for
>> SASL_SSL (which I believe is the common case), seems fine to me.
>>
>> Ismael
>>
>> On Fri, Feb 24, 2017 at 6:18 PM, Christopher Shannon <
>> christopher.l.shan...@gmail.com> wrote:
>>
>> > Rajini, If the override can be dropped for SASL_SSL then I have no
>> problem
>> > with doing this as SASL_SSL External (basically just TLS
>> authentication).
>> > If the configurable callback handlers KIP passes then that would effect
>> > this and one of the callback handlers could be an X509 callback handler.
>> >
>> > Ismael, For why the PrincipalBuilder is not good enough...a custom
>> > PrincipalBuilder would work (would just need to expose the peer
>> > certificates through a new getter but that is simple).  The main reason
>> why
>> > I suggested using JAAS is that is the standard way of plugging in
>> > authentication.  In terms of dual authentication, yes I could have one
>> > listener as SSL and one as SASL.  Or I could just use SASL_SSL and
>> > configure two login modules.
>> >
>> > After thinking about it more maybe the simplest approach would be to
>> just
>> > use a custom PrincipalBuilder along with a small PR to expose the peer
>> > certificates from the SSL handshake.  One thing I didn't like about
>> using a
>> > JAAS module for SSL is that then it is a bit weird because someone might
>> > configure a PrincipalBuilder for returning the Principal but usually
>> that
>> > is the JAAS modules jobs.  So perhaps we should keep it simple and just
>> > rely on the existing PrincinpalBuilder interface so things don't get
>> > confusing.  Thoughts? Would I just close/reject the KIP if I decide to
>> go
>> > this route?
>> >
>> > As a side note, I think it would be a good idea to create a Jira and PR
>> > either way to get rid of the override for ssl.client.auth on SASL_SSL
>> as it
>> > would be good to let the user configure that regardless (i can do the
>> > jira/pr)
>> >
>> > On Fri, Feb 24, 2017 at 9:47 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>> >
>> > > Hi Christopher,
>> > >
>> > > Thanks for the KIP. I have two comments:
>> > >
>> > > 1. Can you please explain in the KIP (maybe in the Rejected
>> Alternatives
>> > > section) why a PrincipalBuilder is not good enough for your use case?
>> > This
>> > > is typically what people use to customise authentication for the TLS
>> > case.
>> > >
>> > > 2. You mention that you have a use case where you need dual
>> > authentication
>> > > (username/password and certificate based). Could you not achieve this
>> via
>> > > two separate listeners? username/password could be via a listener
>> > > configured to use SASL and certificate-based could be via a listener
>> > > configured to use TLS.
>> > >
>> > > Ismael
>> > >
>> > > O

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Sounds good, thanks for the feedback. I will move my KIP to the discarded
section for now as the PrincipalBuilder should be sufficient.  If for some
reason it is not I can revisit this.

Chris

On Fri, Feb 24, 2017 at 1:37 PM Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Christopher,
>
> It is possible to retrieve the certificates already. The Principal returned
> by TransportLayer.peerPrincipal() is a X500Principal if TLS is being used.
> We could add a method to make things nicer, potentially, but just wanted
> you to know that it's possible today.
>
> I am in favour of keeping it simple, if possible. :)
>
> As Rajini said, KIP-103 means users have a way to set the config on a per
> listener basis, so it makes sense to allow users to have both SSL client
> authentication and SASL authentication enabled at the same time if they
> wish. As long as the default remains that SSL client auth is disabled for
> SASL_SSL (which I believe is the common case), seems fine to me.
>
> Ismael
>
> On Fri, Feb 24, 2017 at 6:18 PM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Rajini, If the override can be dropped for SASL_SSL then I have no
> problem
> > with doing this as SASL_SSL External (basically just TLS authentication).
> > If the configurable callback handlers KIP passes then that would effect
> > this and one of the callback handlers could be an X509 callback handler.
> >
> > Ismael, For why the PrincipalBuilder is not good enough...a custom
> > PrincipalBuilder would work (would just need to expose the peer
> > certificates through a new getter but that is simple).  The main reason
> why
> > I suggested using JAAS is that is the standard way of plugging in
> > authentication.  In terms of dual authentication, yes I could have one
> > listener as SSL and one as SASL.  Or I could just use SASL_SSL and
> > configure two login modules.
> >
> > After thinking about it more maybe the simplest approach would be to just
> > use a custom PrincipalBuilder along with a small PR to expose the peer
> > certificates from the SSL handshake.  One thing I didn't like about
> using a
> > JAAS module for SSL is that then it is a bit weird because someone might
> > configure a PrincipalBuilder for returning the Principal but usually that
> > is the JAAS modules jobs.  So perhaps we should keep it simple and just
> > rely on the existing PrincinpalBuilder interface so things don't get
> > confusing.  Thoughts? Would I just close/reject the KIP if I decide to go
> > this route?
> >
> > As a side note, I think it would be a good idea to create a Jira and PR
> > either way to get rid of the override for ssl.client.auth on SASL_SSL as
> it
> > would be good to let the user configure that regardless (i can do the
> > jira/pr)
> >
> > On Fri, Feb 24, 2017 at 9:47 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> >
> > > Hi Christopher,
> > >
> > > Thanks for the KIP. I have two comments:
> > >
> > > 1. Can you please explain in the KIP (maybe in the Rejected
> Alternatives
> > > section) why a PrincipalBuilder is not good enough for your use case?
> > This
> > > is typically what people use to customise authentication for the TLS
> > case.
> > >
> > > 2. You mention that you have a use case where you need dual
> > authentication
> > > (username/password and certificate based). Could you not achieve this
> via
> > > two separate listeners? username/password could be via a listener
> > > configured to use SASL and certificate-based could be via a listener
> > > configured to use TLS.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 21, 2017 at 8:23 PM, Christopher Shannon <
> > > christopher.l.shan...@gmail.com> wrote:
> > >
> > > > Thanks for the feedback Harsha.
> > > >
> > > > Can you clarify what you mean for the use cases for SASL_SSL and
> X509?
> > > My
> > > > proposal is to only have X509 based pluggable authentication for the
> > SSL
> > > > channel and not SASL_SSL.  I suppose you could use X509 credentials
> > with
> > > > SASL_SSL but the authentication mode would probably need to be SASL
> > > > EXTERNAL as the authentication is done by the SSL channel where as
> with
> > > > Kerberos or PLAINTEXT the user is providing credentials.  That's why
> I
> > > > proposed adding it to the SSL channel instead of SASL_SSL.
> > > >
> > > > That being said I guess one option would be to just allow th

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Rajini, If the override can be dropped for SASL_SSL then I have no problem
with doing this as SASL_SSL External (basically just TLS authentication).
If the configurable callback handlers KIP passes then that would effect
this and one of the callback handlers could be an X509 callback handler.

Ismael, For why the PrincipalBuilder is not good enough...a custom
PrincipalBuilder would work (would just need to expose the peer
certificates through a new getter but that is simple).  The main reason why
I suggested using JAAS is that is the standard way of plugging in
authentication.  In terms of dual authentication, yes I could have one
listener as SSL and one as SASL.  Or I could just use SASL_SSL and
configure two login modules.

After thinking about it more maybe the simplest approach would be to just
use a custom PrincipalBuilder along with a small PR to expose the peer
certificates from the SSL handshake.  One thing I didn't like about using a
JAAS module for SSL is that then it is a bit weird because someone might
configure a PrincipalBuilder for returning the Principal but usually that
is the JAAS modules jobs.  So perhaps we should keep it simple and just
rely on the existing PrincinpalBuilder interface so things don't get
confusing.  Thoughts? Would I just close/reject the KIP if I decide to go
this route?

As a side note, I think it would be a good idea to create a Jira and PR
either way to get rid of the override for ssl.client.auth on SASL_SSL as it
would be good to let the user configure that regardless (i can do the
jira/pr)

On Fri, Feb 24, 2017 at 9:47 AM, Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Christopher,
>
> Thanks for the KIP. I have two comments:
>
> 1. Can you please explain in the KIP (maybe in the Rejected Alternatives
> section) why a PrincipalBuilder is not good enough for your use case? This
> is typically what people use to customise authentication for the TLS case.
>
> 2. You mention that you have a use case where you need dual authentication
> (username/password and certificate based). Could you not achieve this via
> two separate listeners? username/password could be via a listener
> configured to use SASL and certificate-based could be via a listener
> configured to use TLS.
>
> Ismael
>
> On Tue, Feb 21, 2017 at 8:23 PM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Thanks for the feedback Harsha.
> >
> > Can you clarify what you mean for the use cases for SASL_SSL and X509?
> My
> > proposal is to only have X509 based pluggable authentication for the SSL
> > channel and not SASL_SSL.  I suppose you could use X509 credentials with
> > SASL_SSL but the authentication mode would probably need to be SASL
> > EXTERNAL as the authentication is done by the SSL channel where as with
> > Kerberos or PLAINTEXT the user is providing credentials.  That's why I
> > proposed adding it to the SSL channel instead of SASL_SSL.
> >
> > That being said I guess one option would be to just allow the use of a
> X509
> > callback handler and don't disable client auth when using SASL_SSL.  Then
> > after login have some way to signal it's EXTERNAL mode so it doesn't do
> any
> > other authentication steps.
> >
> > I have a use case where I need dual authentication (both
> username/password
> > and certificate based) and ether one would work as multiple LoginModules
> > can be chained together.
> >
> > Chris
> >
> > On Tue, Feb 21, 2017 at 3:06 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Hi Chris,
> > >   Thanks for the KIP. Could you also add details/use-cases for
> > > having X509 certificate based authentication in the context SASL_SSL.
> > > The reason that we disabled the SSL auth for SASL_SSL is the intent
> > behind
> > > using SASL auth over SSL encryption and user  can enforce a
> > > role based auth and have wire encryption for data transfer. If users
> just
> > > want SSL based authentication they have option to do so via SSL.
> > > I think we are providing too many options of authentication in SASL_SSL
> > > mode and can be bit confusing.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Feb 21, 2017 at 11:23 AM Christopher Shannon <
> > > christopher.l.shan...@gmail.com> wrote:
> > >
> > > Hi everyone
> > >
> > > I have just created KIP-127 to introduce custom JAAS configuration for
> > the
> > > SSL channel:
> > >
> > > *
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 127%3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
> >

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Does anyone else have any comments or suggestions on this?

On Tue, Feb 21, 2017 at 4:05 PM, Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> I should mention another reason I went with adding this enhancement to the
> SSL channel instead of SASL_SSL is that as you can see from my sample
> commit, I had to delay the JAAS LoginManager from getting loaded until the
> authenticate() call in SslServerAuthenticator in order to make sure that
> the SSL handshake was done because loading the LoginManager does the actual
> login() call and requires the X509 callback handler.
>
> The SASL_SSL implementation loads the LoginManager during the configure in
> SaslChannelBuilder which is too early as the X509 credentials won't be
> available yet without the handshake being completed so this would require
> some refactoring to get to this to work properly and load at the right time.
>
> On Tue, Feb 21, 2017 at 3:44 PM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
>> As a follow up to my previous post, EXTERNAL could be added to the list
>> of mechanisms supported with the existing property: sasl.enabled.mechanisms
>> so I think this could also be achieved with SASL_SSL.  If EXTERNAL is used
>> then it would not disable the client certificate from being required.
>>
>> So I can go either way on this, I can update my KIP to allow X509
>> authentication with SASL_SSL through the EXTERNAL mechanism or keep the
>> proposal as is for the SSL channel based on what everyone thinks.
>>
>> On Tue, Feb 21, 2017 at 3:23 PM, Christopher Shannon <
>> christopher.l.shan...@gmail.com> wrote:
>>
>>> Thanks for the feedback Harsha.
>>>
>>> Can you clarify what you mean for the use cases for SASL_SSL and X509?
>>> My proposal is to only have X509 based pluggable authentication for the SSL
>>> channel and not SASL_SSL.  I suppose you could use X509 credentials with
>>> SASL_SSL but the authentication mode would probably need to be SASL
>>> EXTERNAL as the authentication is done by the SSL channel where as with
>>> Kerberos or PLAINTEXT the user is providing credentials.  That's why I
>>> proposed adding it to the SSL channel instead of SASL_SSL.
>>>
>>> That being said I guess one option would be to just allow the use of a
>>> X509 callback handler and don't disable client auth when using SASL_SSL.
>>> Then after login have some way to signal it's EXTERNAL mode so it doesn't
>>> do any other authentication steps.
>>>
>>> I have a use case where I need dual authentication (both
>>> username/password and certificate based) and ether one would work as
>>> multiple LoginModules can be chained together.
>>>
>>> Chris
>>>
>>> On Tue, Feb 21, 2017 at 3:06 PM, Harsha Chintalapani <ka...@harsha.io>
>>> wrote:
>>>
>>>> Hi Chris,
>>>>   Thanks for the KIP. Could you also add details/use-cases for
>>>> having X509 certificate based authentication in the context SASL_SSL.
>>>> The reason that we disabled the SSL auth for SASL_SSL is the intent
>>>> behind
>>>> using SASL auth over SSL encryption and user  can enforce a
>>>> role based auth and have wire encryption for data transfer. If users
>>>> just
>>>> want SSL based authentication they have option to do so via SSL.
>>>> I think we are providing too many options of authentication in SASL_SSL
>>>> mode and can be bit confusing.
>>>>
>>>> Thanks,
>>>> Harsha
>>>>
>>>>
>>>> On Tue, Feb 21, 2017 at 11:23 AM Christopher Shannon <
>>>> christopher.l.shan...@gmail.com> wrote:
>>>>
>>>> Hi everyone
>>>>
>>>> I have just created KIP-127 to introduce custom JAAS configuration for
>>>> the
>>>> SSL channel:
>>>>
>>>> *
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%3A
>>>> +Pluggable+JAAS+LoginModule+configuration+for+SSL
>>>> <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%3A
>>>> +Pluggable+JAAS+LoginModule+configuration+for+SSL
>>>> >*
>>>>
>>>> The idea here is to be able to do custom authentication based off of a
>>>> user's X509 credentials in addition to the SSL handshake.
>>>>
>>>> I have created a rough draft of a commit to give an idea of what my
>>>> plan is
>>>> which matches the KIP:
>>>> https://github.com/cshannon/kafka/tree/KAFKA-4784
>>>>
>>>> It still needs some work (needs more tests for example) but I wanted to
>>>> get
>>>> some feedback before I went any farther on this and do a pull request.
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>
>>>
>>
>

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

I should mention another reason I went with adding this enhancement to the
SSL channel instead of SASL_SSL is that as you can see from my sample
commit, I had to delay the JAAS LoginManager from getting loaded until the
authenticate() call in SslServerAuthenticator in order to make sure that
the SSL handshake was done because loading the LoginManager does the actual
login() call and requires the X509 callback handler.

The SASL_SSL implementation loads the LoginManager during the configure in
SaslChannelBuilder which is too early as the X509 credentials won't be
available yet without the handshake being completed so this would require
some refactoring to get to this to work properly and load at the right time.

On Tue, Feb 21, 2017 at 3:44 PM, Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> As a follow up to my previous post, EXTERNAL could be added to the list of
> mechanisms supported with the existing property: sasl.enabled.mechanisms
> so I think this could also be achieved with SASL_SSL.  If EXTERNAL is used
> then it would not disable the client certificate from being required.
>
> So I can go either way on this, I can update my KIP to allow X509
> authentication with SASL_SSL through the EXTERNAL mechanism or keep the
> proposal as is for the SSL channel based on what everyone thinks.
>
> On Tue, Feb 21, 2017 at 3:23 PM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
>> Thanks for the feedback Harsha.
>>
>> Can you clarify what you mean for the use cases for SASL_SSL and X509?
>> My proposal is to only have X509 based pluggable authentication for the SSL
>> channel and not SASL_SSL.  I suppose you could use X509 credentials with
>> SASL_SSL but the authentication mode would probably need to be SASL
>> EXTERNAL as the authentication is done by the SSL channel where as with
>> Kerberos or PLAINTEXT the user is providing credentials.  That's why I
>> proposed adding it to the SSL channel instead of SASL_SSL.
>>
>> That being said I guess one option would be to just allow the use of a
>> X509 callback handler and don't disable client auth when using SASL_SSL.
>> Then after login have some way to signal it's EXTERNAL mode so it doesn't
>> do any other authentication steps.
>>
>> I have a use case where I need dual authentication (both
>> username/password and certificate based) and ether one would work as
>> multiple LoginModules can be chained together.
>>
>> Chris
>>
>> On Tue, Feb 21, 2017 at 3:06 PM, Harsha Chintalapani <ka...@harsha.io>
>> wrote:
>>
>>> Hi Chris,
>>>   Thanks for the KIP. Could you also add details/use-cases for
>>> having X509 certificate based authentication in the context SASL_SSL.
>>> The reason that we disabled the SSL auth for SASL_SSL is the intent
>>> behind
>>> using SASL auth over SSL encryption and user  can enforce a
>>> role based auth and have wire encryption for data transfer. If users just
>>> want SSL based authentication they have option to do so via SSL.
>>> I think we are providing too many options of authentication in SASL_SSL
>>> mode and can be bit confusing.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>> On Tue, Feb 21, 2017 at 11:23 AM Christopher Shannon <
>>> christopher.l.shan...@gmail.com> wrote:
>>>
>>> Hi everyone
>>>
>>> I have just created KIP-127 to introduce custom JAAS configuration for
>>> the
>>> SSL channel:
>>>
>>> *
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%3A
>>> +Pluggable+JAAS+LoginModule+configuration+for+SSL
>>> <
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%3A
>>> +Pluggable+JAAS+LoginModule+configuration+for+SSL
>>> >*
>>>
>>> The idea here is to be able to do custom authentication based off of a
>>> user's X509 credentials in addition to the SSL handshake.
>>>
>>> I have created a rough draft of a commit to give an idea of what my plan
>>> is
>>> which matches the KIP:
>>> https://github.com/cshannon/kafka/tree/KAFKA-4784
>>>
>>> It still needs some work (needs more tests for example) but I wanted to
>>> get
>>> some feedback before I went any farther on this and do a pull request.
>>>
>>> Thanks,
>>> Chris
>>>
>>
>>
>

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

As a follow up to my previous post, EXTERNAL could be added to the list of
mechanisms supported with the existing property: sasl.enabled.mechanisms so
I think this could also be achieved with SASL_SSL.  If EXTERNAL is used
then it would not disable the client certificate from being required.

So I can go either way on this, I can update my KIP to allow X509
authentication with SASL_SSL through the EXTERNAL mechanism or keep the
proposal as is for the SSL channel based on what everyone thinks.

On Tue, Feb 21, 2017 at 3:23 PM, Christopher Shannon <
christopher.l.shan...@gmail.com> wrote:

> Thanks for the feedback Harsha.
>
> Can you clarify what you mean for the use cases for SASL_SSL and X509?  My
> proposal is to only have X509 based pluggable authentication for the SSL
> channel and not SASL_SSL.  I suppose you could use X509 credentials with
> SASL_SSL but the authentication mode would probably need to be SASL
> EXTERNAL as the authentication is done by the SSL channel where as with
> Kerberos or PLAINTEXT the user is providing credentials.  That's why I
> proposed adding it to the SSL channel instead of SASL_SSL.
>
> That being said I guess one option would be to just allow the use of a
> X509 callback handler and don't disable client auth when using SASL_SSL.
> Then after login have some way to signal it's EXTERNAL mode so it doesn't
> do any other authentication steps.
>
> I have a use case where I need dual authentication (both username/password
> and certificate based) and ether one would work as multiple LoginModules
> can be chained together.
>
> Chris
>
> On Tue, Feb 21, 2017 at 3:06 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
>> Hi Chris,
>>   Thanks for the KIP. Could you also add details/use-cases for
>> having X509 certificate based authentication in the context SASL_SSL.
>> The reason that we disabled the SSL auth for SASL_SSL is the intent behind
>> using SASL auth over SSL encryption and user  can enforce a
>> role based auth and have wire encryption for data transfer. If users just
>> want SSL based authentication they have option to do so via SSL.
>> I think we are providing too many options of authentication in SASL_SSL
>> mode and can be bit confusing.
>>
>> Thanks,
>> Harsha
>>
>>
>> On Tue, Feb 21, 2017 at 11:23 AM Christopher Shannon <
>> christopher.l.shan...@gmail.com> wrote:
>>
>> Hi everyone
>>
>> I have just created KIP-127 to introduce custom JAAS configuration for the
>> SSL channel:
>>
>> *
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%
>> 3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%
>> 3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
>> >*
>>
>> The idea here is to be able to do custom authentication based off of a
>> user's X509 credentials in addition to the SSL handshake.
>>
>> I have created a rough draft of a commit to give an idea of what my plan
>> is
>> which matches the KIP:
>> https://github.com/cshannon/kafka/tree/KAFKA-4784
>>
>> It still needs some work (needs more tests for example) but I wanted to
>> get
>> some feedback before I went any farther on this and do a pull request.
>>
>> Thanks,
>> Chris
>>
>
>

Re: [DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Thanks for the feedback Harsha.

Can you clarify what you mean for the use cases for SASL_SSL and X509?  My
proposal is to only have X509 based pluggable authentication for the SSL
channel and not SASL_SSL.  I suppose you could use X509 credentials with
SASL_SSL but the authentication mode would probably need to be SASL
EXTERNAL as the authentication is done by the SSL channel where as with
Kerberos or PLAINTEXT the user is providing credentials.  That's why I
proposed adding it to the SSL channel instead of SASL_SSL.

That being said I guess one option would be to just allow the use of a X509
callback handler and don't disable client auth when using SASL_SSL.  Then
after login have some way to signal it's EXTERNAL mode so it doesn't do any
other authentication steps.

I have a use case where I need dual authentication (both username/password
and certificate based) and ether one would work as multiple LoginModules
can be chained together.

Chris

On Tue, Feb 21, 2017 at 3:06 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Hi Chris,
>   Thanks for the KIP. Could you also add details/use-cases for
> having X509 certificate based authentication in the context SASL_SSL.
> The reason that we disabled the SSL auth for SASL_SSL is the intent behind
> using SASL auth over SSL encryption and user  can enforce a
> role based auth and have wire encryption for data transfer. If users just
> want SSL based authentication they have option to do so via SSL.
> I think we are providing too many options of authentication in SASL_SSL
> mode and can be bit confusing.
>
> Thanks,
> Harsha
>
>
> On Tue, Feb 21, 2017 at 11:23 AM Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> Hi everyone
>
> I have just created KIP-127 to introduce custom JAAS configuration for the
> SSL channel:
>
> *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 127%3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 127%3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
> >*
>
> The idea here is to be able to do custom authentication based off of a
> user's X509 credentials in addition to the SSL handshake.
>
> I have created a rough draft of a commit to give an idea of what my plan is
> which matches the KIP:
> https://github.com/cshannon/kafka/tree/KAFKA-4784
>
> It still needs some work (needs more tests for example) but I wanted to get
> some feedback before I went any farther on this and do a pull request.
>
> Thanks,
> Chris
>

[DISCUSS] KIP-127: Pluggable JAAS LoginModule configuration for SSL

[Christopher Shannon]

Hi everyone

I have just created KIP-127 to introduce custom JAAS configuration for the
SSL channel:

*https://cwiki.apache.org/confluence/display/KAFKA/KIP-127%3A+Pluggable+JAAS+LoginModule+configuration+for+SSL
*

The idea here is to be able to do custom authentication based off of a
user's X509 credentials in addition to the SSL handshake.

I have created a rough draft of a commit to give an idea of what my plan is
which matches the KIP:
https://github.com/cshannon/kafka/tree/KAFKA-4784

It still needs some work (needs more tests for example) but I wanted to get
some feedback before I went any farther on this and do a pull request.

Thanks,
Chris

Re: Request Wiki edit permission for cshannon

[Christopher Shannon]

Awesome, thanks.

On Fri, Feb 17, 2017 at 2:30 PM, Guozhang Wang <wangg...@gmail.com> wrote:

> Done.
>
> Guozhang
>
> On Fri, Feb 17, 2017 at 10:20 AM, Christopher Shannon <
> christopher.l.shan...@gmail.com> wrote:
>
> > Hi,
> >
> > I would like to contribute a KIP.  Can someone give me access to the wiki
> > to create pages?
> >
> > Thanks,
> > Chris
> >
>
>
>
> --
> -- Guozhang
>

Request Wiki edit permission for cshannon

[Christopher Shannon]

Hi,

I would like to contribute a KIP.  Can someone give me access to the wiki
to create pages?

Thanks,
Chris