Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Luke Chen]

Hi Tina,

Thanks for the KIP to fix the issue.

Some comments:
1. In the compatibility section, you said:
`By default, the new setting will be set to false so it does not change the
current behaviour.`

I'm confused, what is the config we need to set to `false` to avoid
breaking compatibility?
All I can see is there is one new config introduced:
use.incremental.alter.configs, and default to "requested".
Does that mean it'll change current behavior?
If so, I think we should make it clear in the compatibility section about
what will be changed after this KIP.

2. It looks like you're going to introduce a new method in the existing
interface. Could you follow the pattern in other KIP (ex: KIP-888
)
to list the javadoc and the method name together?

Thank you.
Luke


On Fri, Feb 3, 2023 at 11:46 PM Chris Egerton 
wrote:

> Hi Tina,
>
> Thanks for the KIP! I recently ran into this exact issue and it's great to
> see a fix being proposed. I have a few small comments but overall this
> looks good:
>
> 1) The current logic for syncing topic configs in MM2 is basically
> fire-and-forget; all we do is log a warning message [1] if an attempt
> fails. When "use.incremental.alter.configs" is set to "requested", we'll
> need to know whether attempts using the incremental API succeed or not, and
> then adjust our behavior accordingly. Will the plan here be to block on the
> success/failure of the first request before sending any more, or will we
> just switch over to the legacy API as soon as any request fails due to
> targeting an incompatible broker, possibly after multiple requests with the
> new API have been sent or at least queued up in the admin client?
>
> 2) We don't replicate default properties from the source cluster right now
> [2].
> Won't making ConfigPropertyFilter::shouldReplicateSourceDefault return true
> by default change that behavior? If so, what's the motivation for favoring
> defaults from the source cluster over defaults for the target cluster,
> especially given that users may already be relying on the opposite?
>
> 3) Nit: IMO the parts in the "proposed changes" section that detail changes
> to internal classes aren't really necessary since they're not relevant to
> user-facing behavior and the classes/methods described in them might change
> between now and when the PR for the KIP gets opened/reviewed/merged. I
> think the only points that need to be in the KIP are the ones beginning
> with "Extend ConfigPropertyFilter class", "Add a new configuration setting
> to MirrorMaker", and "From Kafka 4.0" (which itself can just describe the
> broker APIs that are used by MM2 in general, without referring to the
> specific name of the method in MM2 that will call them).
>
> [1] -
>
> https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L429
>
> [2] -
>
> https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L460
>
> Thanks again for the KIP!
>
> Cheers,
>
> Chris
>
> On Wed, Jan 25, 2023 at 10:44 AM Gantigmaa Selenge 
> wrote:
>
> > Hi,
> >
> > If there are no further comments on the KIP, I will start a vote on it.
> >
> > Regards,
> >
> >
> > On Mon, Jan 16, 2023 at 11:14 AM Gantigmaa Selenge 
> > wrote:
> >
> > > Thanks everyone.
> > >
> > > I took the suggestions and updated the KIP accordingly. Please let me
> > know
> > > if there is anything else I could improve on.
> > >
> > > Regards,
> > > Tina
> > >
> > > On Sun, Jan 15, 2023 at 10:24 PM Ismael Juma 
> wrote:
> > >
> > >> Hi Tina,
> > >>
> > >> See below.
> > >>
> > >> On Wed, Jan 11, 2023 at 3:03 AM Gantigmaa Selenge <
> gsele...@redhat.com>
> > >> wrote:
> > >>
> > >> > I do like this idea, however when it's set to required, I wasn't
> sure
> > >> how
> > >> > the mirrormaker should have. It's probably not a great experience if
> > >> > mirrormaker starts crashing at some point after it's already running
> > >> due to
> > >> > an incompatible broker version.
> > >> >
> > >>
> > >> This would only happen if the user explicitly requests the strict
> > required
> > >> ("non fallback") mode. There are many reasons why one may want this:
> say
> > >> you want to be sure that your system is not susceptible to the
> > >> "alterConfigs" problems or you want to write a test that fails if
> > >> "alterConfigs' is used.
> > >>
> > >>
> > >> > If the incrementalAlterConfig request fails because the target
> cluster
> > >> is
> > >> > running an older version, then we could log a WARN message that says
> > >> > something like  "The config to use incrementalAlterConfig API for
> > >> syncing
> > >> > topic configurations has been set to true however target cluster 

[GitHub] [kafka-site] mumrah opened a new pull request, #486: Update website for 3.4.0 release

[via GitHub]

mumrah opened a new pull request, #486:
URL: https://github.com/apache/kafka-site/pull/486

   This PR updates the "33" template references to "34" for the 3.4.0 release. 
It also adds the released artifacts to the downloads page and points the 
documentation page at the 3.4 documentation page


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-14686) MockAdminClient.createTopics does not provide TopicMetadataAndConfig

[Kirk True (Jira)]

Kirk True created KAFKA-14686:
-

 Summary: MockAdminClient.createTopics does not provide 
TopicMetadataAndConfig
 Key: KAFKA-14686
 URL: https://issues.apache.org/jira/browse/KAFKA-14686
 Project: Kafka
  Issue Type: Bug
  Components: admin, clients, unit tests
Reporter: Kirk True
Assignee: Kirk True


[Line 386 of 
{{MockAdminClient}}|https://github.com/apache/kafka/blame/trunk/clients/src/test/java/org/apache/kafka/clients/admin/MockAdminClient.java#L386]
 does this:
{quote}{{future.complete(null);}}
{quote}
It seems like we should be creating a {{TopicMetadataAndConfig}} instance and 
passing that in instead so that its available to the unit test caller.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14685) TierStateMachine interface for building remote aux log

[Matthew Wong (Jira)]

Matthew Wong created KAFKA-14685:


 Summary: TierStateMachine interface for building remote aux log
 Key: KAFKA-14685
 URL: https://issues.apache.org/jira/browse/KAFKA-14685
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Reporter: Matthew Wong


To help with https://issues.apache.org/jira/browse/KAFKA-13560 , we can 
introduce an interface to manage state transitions of building the remote aux 
log asynchronously



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #75

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1560

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14684) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest

[Hector Geraldino (Jira)]

Hector Geraldino created KAFKA-14684:


 Summary: Replace EasyMock and PowerMock with Mockito in 
WorkerSinkTaskThreadedTest
 Key: KAFKA-14684
 URL: https://issues.apache.org/jira/browse/KAFKA-14684
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Chris Egerton]

Hi Mickael,

I agree that adding a getter method for Monitorable isn't great. A few
alternatives come to mind:

1. Introduce a new ConfiguredInstance (name subject to change) interface
that wraps an instance of type T, but also contains a getter method for any
PluginMetrics instances that the plugin was instantiated with (which may
return null either if no PluginMetrics instance could be created for the
plugin, or if it did not implement the Monitorable interface). This can be
the return type of the new AbstractConfig::getConfiguredInstance variants.
It would give us room to move forward with other plugin-for-your-plugin
style interfaces without cluttering things up with getter methods. We could
even add a close method to this interface which would handle cleanup of all
extra resources allocated for the plugin by the runtime, and even possibly
the plugin itself.

2. Break out the instantiation logic into two separate steps. The first
step, creating a PluginMetrics instance, can be either private or public
API. The second step, providing that PluginMetrics instance to a
newly-created object, can be achieved with a small tweak of the proposed
new methods for the AbstractConfig class; instead of accepting a Metrics
instance, they would now accept a PluginMetrics instance. For the first
step, we might even introduce a new CloseablePluginMetrics interface which
would be the return type of whatever method we use to create the
PluginMetrics instance. We can track that CloseablePluginMetrics instance
in tandem with the plugin it applies to, and close it when we're done with
the plugin.

I know that this adds some complexity to the API design and some
bookkeeping responsibilities for our implementation, but I can't shake the
feeling that if we don't feel comfortable taking on the responsibility to
clean up these resources ourselves, it's not really fair to ask users to
handle it for us instead. And with the case of Connect, sometimes Connector
or Task instances that are scheduled for shutdown block for a while, at
which point we abandon them and bring up new instances in their place; it'd
be nice to have a way to forcibly clear out all the metrics allocated by
that Connector or Task instance before bringing up a new one, in order to
prevent issues due to naming conflicts.

Regardless, and whether or not it ends up being relevant to this KIP, I'd
love to see a new Converter::close method. It's irked me for quite a while
that we don't have one already.

Cheers,

Chris

On Mon, Feb 6, 2023 at 1:50 PM Mickael Maison 
wrote:

> Hi Chris,
>
> I envisioned plugins to be responsible for closing the PluginMetrics
> instance. This is mostly important for Connect connector plugins as
> they can be closed while the runtime keeps running (and keeps its
> Metrics instance). As far as I can tell, other plugins should only be
> closed when their runtime closes, so we should not be leaking metrics
> even if those don't explicitly call close().
>
> For Connect plugin, as you said, it would be nice to automatically
> close their associated PluginMetrics rather than relying on user
> logic. The issue is that with the current API there's no way to
> retrieve the PluginMetrics instance once it's passed to the plugin.
> I'm not super keen on having a getter method on the Monitorable
> interface and tracking PluginMetrics instances associated with each
> plugin would require a lot of changes. I just noticed Converter does
> not have a close() method so it's problematic for that type of plugin.
> The other Connect plugins all have close() or stop() methods. I wonder
> if the simplest is to make Converter extend Closeable. WDYT?
>
> Thanks,
> Mickael
>
> On Mon, Feb 6, 2023 at 6:39 PM Mickael Maison 
> wrote:
> >
> > Hi Yash,
> >
> > I added a sentence to the sensor() method mentioning the sensor name
> > must only be unique per plugin. Regarding having getters for sensors
> > and metrics I considered this not strictly necessary as I expect the
> > metrics logic in most plugins to be relatively simple. If you have a
> > use case that would benefit from these methods, let me know I will
> > reconsider.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Fri, Feb 3, 2023 at 9:16 AM Yash Mayya  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the updates.
> > >
> > > > the PluginMetrics implementation will append a
> > > > suffix to sensor names to unique identify
> > > > the plugin (based on the class name and tags).
> > >
> > > Can we call this out explicitly in the KIP, since it is important to
> avoid
> > > clashes in sensor naming? Also, should we allow plugins to retrieve
> sensors
> > > from `PluginMetrics` if we can check / verify that they own the sensor
> > > (based on the suffix)?
> > >
> > > Other than the above minor points, this looks good to me now!
> > >
> > > Thanks,
> > > Yash
> > >
> > > On Fri, Feb 3, 2023 at 2:29 AM Chris Egerton 
> > > wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > This is looking great. I have one small 

[jira] [Created] (KAFKA-14683) Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskTest

[Hector Geraldino (Jira)]

Hector Geraldino created KAFKA-14683:


 Summary: Replace EasyMock and PowerMock with Mockito in 
WorkerSinkTaskTest
 Key: KAFKA-14683
 URL: https://issues.apache.org/jira/browse/KAFKA-14683
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Hector Geraldino
Assignee: Hector Geraldino






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14682) Unused stubbings are not reported by Mockito during CI builds

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14682:
-

 Summary: Unused stubbings are not reported by Mockito during CI 
builds
 Key: KAFKA-14682
 URL: https://issues.apache.org/jira/browse/KAFKA-14682
 Project: Kafka
  Issue Type: Test
  Components: unit tests
Reporter: Chris Egerton


We've started using [strict 
stubbing|https://javadoc.io/static/org.mockito/mockito-core/4.6.1/org/mockito/junit/MockitoJUnitRunner.StrictStubs.html]
 for unit tests written with Mockito, which is supposed to automatically fail 
tests when they set up mock expectations that go unused.

However, these failures are not reported during Jenkins builds, even if they 
are reported when building/testing locally.

In at least one case, this difference appears to be because our [Jenkins 
build|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/Jenkinsfile#L32-L35]
 uses the custom {{unitTest}} and {{integrationTest}} tasks defined in the 
project's [Gradle build 
file|https://github.com/apache/kafka/blob/6d11261d5deaca300e273bebe309f9e4f814f815/build.gradle#L452-L543],
 instead of the {{test}} task.

It's possible that, because the custom test tasks filter out some tests from 
running, Mockito does not check for unnecessary stubbings in order to avoid 
incorrectly failing tests that set up mocks in, e.g., a {{@BeforeEach}} method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #74

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14681) Refreshing Kerberos TGT is not using CallbackHandler (causing failure to refresh)

[Vlad D. (Jira)]

Vlad D. created KAFKA-14681:
---

 Summary: Refreshing Kerberos TGT is not using CallbackHandler 
(causing failure to refresh)
 Key: KAFKA-14681
 URL: https://issues.apache.org/jira/browse/KAFKA-14681
 Project: Kafka
  Issue Type: Bug
  Components: security
Reporter: Vlad D.


The SASL + Kerberos authentication in KerberosLogin.java class, when obtaining 
Kerberos TGT, makes use of the client-provided callback handler. This is a 
must-have when the security configuration is not default.

However, the same code, when it is time to renew the Kerberos TGT ticket, 
ignores the provided ticket. That works OK for default configuration (JAAS 
configuration, Kerberos config and keytab are available).

But when the security configuration sources are custom, and the default 
Kerberos code is not supporting them, the callback is to be used to obtain the 
configuration properties.

A fix is done to pass the same callback handler in KerberosLogin::reLogin and 
store the callback handler in the super class AbstractLogin, similar to 
contextName and configuration.

The fix is in PR [https://github.com/apache/kafka/pull/13081]

It is tested in our SFT environments and works fine.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13972) Reassignment cancellation causes stray replicas

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13972.
-
Resolution: Fixed

> Reassignment cancellation causes stray replicas
> ---
>
> Key: KAFKA-13972
> URL: https://issues.apache.org/jira/browse/KAFKA-13972
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.4.1
>
>
> A stray replica is one that is left behind on a broker after the partition 
> has been reassigned to other brokers or the partition has been deleted. We 
> found one case where this can happen is after a cancelled reassignment. When 
> a reassignment is cancelled, the controller sends `StopReplica` requests to 
> any of the adding replicas, but it does not necessarily bump the leader 
> epoch. Following 
> [KIP-570|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-570%3A+Add+leader+epoch+in+StopReplicaRequest],]
>  brokers will ignore `StopReplica` requests if the leader epoch matches the 
> current partition leader epoch. So we need to bump the epoch whenever we need 
> to ensure that `StopReplica` will be received.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Chris,

I envisioned plugins to be responsible for closing the PluginMetrics
instance. This is mostly important for Connect connector plugins as
they can be closed while the runtime keeps running (and keeps its
Metrics instance). As far as I can tell, other plugins should only be
closed when their runtime closes, so we should not be leaking metrics
even if those don't explicitly call close().

For Connect plugin, as you said, it would be nice to automatically
close their associated PluginMetrics rather than relying on user
logic. The issue is that with the current API there's no way to
retrieve the PluginMetrics instance once it's passed to the plugin.
I'm not super keen on having a getter method on the Monitorable
interface and tracking PluginMetrics instances associated with each
plugin would require a lot of changes. I just noticed Converter does
not have a close() method so it's problematic for that type of plugin.
The other Connect plugins all have close() or stop() methods. I wonder
if the simplest is to make Converter extend Closeable. WDYT?

Thanks,
Mickael

On Mon, Feb 6, 2023 at 6:39 PM Mickael Maison  wrote:
>
> Hi Yash,
>
> I added a sentence to the sensor() method mentioning the sensor name
> must only be unique per plugin. Regarding having getters for sensors
> and metrics I considered this not strictly necessary as I expect the
> metrics logic in most plugins to be relatively simple. If you have a
> use case that would benefit from these methods, let me know I will
> reconsider.
>
> Thanks,
> Mickael
>
>
> On Fri, Feb 3, 2023 at 9:16 AM Yash Mayya  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the updates.
> >
> > > the PluginMetrics implementation will append a
> > > suffix to sensor names to unique identify
> > > the plugin (based on the class name and tags).
> >
> > Can we call this out explicitly in the KIP, since it is important to avoid
> > clashes in sensor naming? Also, should we allow plugins to retrieve sensors
> > from `PluginMetrics` if we can check / verify that they own the sensor
> > (based on the suffix)?
> >
> > Other than the above minor points, this looks good to me now!
> >
> > Thanks,
> > Yash
> >
> > On Fri, Feb 3, 2023 at 2:29 AM Chris Egerton 
> > wrote:
> >
> > > Hi Mickael,
> > >
> > > This is looking great. I have one small question left but I do not 
> > > consider
> > > it a blocker.
> > >
> > > What is the intended use case for PluginMetrics::close? To me at least, it
> > > implies that plugin developers will be responsible for invoking that 
> > > method
> > > themselves in order to clean up metrics that they've created, but wouldn't
> > > we want the runtime (i.e., KafkaProducer class, Connect framework, etc.) 
> > > to
> > > handle that automatically when the resource that the plugin applies to is
> > > closed?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Jan 26, 2023 at 10:22 AM Mickael Maison 
> > > wrote:
> > >
> > > > Hi Yash,
> > > >
> > > > 1) To avoid conflicts with other sensors, the PluginMetrics
> > > > implementation will append a suffix to sensor names to unique identify
> > > > the plugin (based on the class name and tags). Also I changed the
> > > > semantics of the sensor() method to only create sensors (originally it
> > > > was get or create). If a sensor with the same name already exists, the
> > > > method will throw.
> > > > 2) Tags will be automatically added to metrics and sensors to unique
> > > > identify the plugin. For Connect plugins, the connector name, task id
> > > > and alias can be added if available. The class implementing
> > > > PluginMetrics will be similar to ConnectMetrics, as in it will provide
> > > > a simplified API wrapping Metrics. I'm planning to use PluginMetrics
> > > > for Connect plugin too and should not need to interact with
> > > > ConnectMetrics.
> > > > 3) Right, I fixed the last rejected alternative.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Thu, Jan 26, 2023 at 4:04 PM Mickael Maison  > > >
> > > > wrote:
> > > > >
> > > > > Hi Federico,
> > > > >
> > > > > - The metricName() method does not register anything, it just builds a
> > > > > MetricName instance which is just a container holding a name, group,
> > > > > description and tags for a metric. Each time it is called, it returns
> > > > > a new instance. If called with the same arguments, the returned value
> > > > > will be equal.
> > > > > - Initially I just copied the API of Metrics. I made some small
> > > > > changes so the metric and sensor methods are a bit more similar
> > > > > - Good catch! I fixed the example.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Thu, Jan 26, 2023 at 3:54 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > 1) I updated the KIP to only mention the interface.
> > > > > > 2) This was a mistake. I've added ReplicationPolicy to the list of
> > > > plugins.
> > > > > >
> > > > > > 

[jira] [Created] (KAFKA-14680) Gradle version upgrade 7 -->> 8

[Jira]

Dejan Stojadinović created KAFKA-14680:
--

 Summary: Gradle version upgrade 7 -->> 8
 Key: KAFKA-14680
 URL: https://issues.apache.org/jira/browse/KAFKA-14680
 Project: Kafka
  Issue Type: Improvement
  Components: build
Reporter: Dejan Stojadinović
Assignee: Dejan Stojadinović


*Gradle 8.0.0-RC3 release notes* (note: final 8.0 version is to be released 
soon):
 * [https://github.com/gradle/gradle/releases/tag/v8.0.0-RC3]
 * [https://docs.gradle.org/8.0-rc-3/release-notes.html]

*Upgrade notes:* 
[https://docs.gradle.org/8.0-rc-3/userguide/upgrading_version_7.html#changes_8.0]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi David,

Thank you for your suggestions and for having a look at this KIP.

1. Yes, that should be OK. I have updated the section
"Migrating a cluster in ZK mode running with JBOD" to reflect this.

2. I've updated the motivation section to state that.

Best,

--
Igor

[jira] [Resolved] (KAFKA-14663) High throughput topics can starve low-throughput MM2 offset syncs

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14663.
---
Resolution: Duplicate

> High throughput topics can starve low-throughput MM2 offset syncs
> -
>
> Key: KAFKA-14663
> URL: https://issues.apache.org/jira/browse/KAFKA-14663
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 3.1.0, 3.0.0, 3.3.0, 3.4.0, 3.5.0
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
>
> In MM2, a semaphore is used to throttle the number of offset syncs written to 
> the offset-syncs topic. If too many offset writes are requested (for example, 
> from high-throughput topics) then some are silently dropped and never 
> retried. This is acceptable for a single topic-partition, where a later 
> record may re-trigger the offset-sync and write the sync successfully.
> However, if there is a large variance between throughput in the topics 
> emitted by an MM2 instance, it is possible for high-throughput topics to 
> trigger many offset syncs, and cause the offset-syncs for a co-located 
> low-throughput topic to be unfairly dropped.
> This can cause the downstream offsets for the starved topic to lag behind 
> significantly, or be prevented completely.
> Instead, we should have some sort of fairness mechanism where low-thoughput 
> topics are given similar priority to high-throughput topics in producing 
> offset syncs, and cause excess sync messages from high-throughput topics to 
> be dropped instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

[Chris Egerton]

Hi Yash,

I've had some time to think on this KIP and I think I'm in agreement about
not blocking it on an official compatibility library or adding the "ack"
API for sink records.

I only have two more thoughts:

1. Because it is possible to manipulate sink record partitions and offsets
with the current API we provide for transformations, I still believe
methods should be added to the SinkRecord class to expose the original
partition and offset, not just the original topic. The additional cognitive
burden from these two methods is going to be minimal anyways; once users
understand the difference between the transformed topic name and the
original one, it's going to be trivial for them to understand how that same
difference applies for partitions and offsets. It's not enough to scan the
set of SMTs provided out of the box with Connect, ones developed by
Confluent, or even everything available on GitHub, since there may be
closed-source projects out there that rely on this ability. One potential
use case could be re-routing partitions between Kafka and some other
sharded system.

2. We still have to address the SinkTask::open [1] and SinkTask::close [2]
methods. If a connector writes to the external system using the transformed
topic partitions it reads from Kafka, then it's possible for the connector
to lazily instantiate writers for topic partitions as it encounters them
from records provided in SinkTask::put. However, connectors also need a way
to de-allocate those writers (and the resources used by them) over time,
which they can't do as easily. One possible approach here is to overload
SinkTask::open and SinkTask::close with variants that distinguish between
transformed and original topic partitions, and default to invoking the
existing methods with just the original topic partitions. We would then
have several options for how the Connect runtime can invoke these methods,
but in general, an approach that guarantees that tasks are notified of
transformed topic partitions in SinkTask::open before any records for that
partition are given to it in SinkTask::put, and makes a best-effort attempt
to close transformed topic partitions that appear to no longer be in use
based on some eviction policy, would probably be sufficient.

[1] -
https://kafka.apache.org/33/javadoc/org/apache/kafka/connect/sink/SinkTask.html#open(java.util.Collection)
[2] -
https://kafka.apache.org/33/javadoc/org/apache/kafka/connect/sink/SinkTask.html#close(java.util.Collection)

Cheers,

Chris

On Sat, Nov 5, 2022 at 5:46 AM Yash Mayya  wrote:

> Hi Chris,
>
> Thanks a lot for your inputs!
>
> > would provide a simple, clean interface for developers to determine
> > which features are supported by the version of the Connect runtime
> > that their plugin has been deployed onto
>
> I do like the idea of having such a public compatibility library - I think
> it would remove a lot of restrictions from framework development if it were
> to be widely adopted.
>
> > we might consider adding an API to "ack" sink records
>
> I agree that this does seem like a more intuitive and clean API, but I'm
> concerned about the backward compatibility headache we'd be imposing on all
> existing sink connectors. Connector developers will have to maintain two
> separate ways of doing offset management if they want to use the new API
> but continue supporting older versions of Kafka Connect.
>
> For now, I've reverted the KIP to the previous iteration which proposed the
> addition of a new `SinkRecord` method to obtain the original Kafka topic
> pre-transformation. One thing to note is that I've removed the method for
> obtaining the original Kafka partition after a cursory search showed that
> use cases for partition modifying SMTs are primarily on the source
> connector side.
>
> Thanks,
> Yash
>
> On Tue, Nov 1, 2022 at 9:22 PM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > I have more comments I'd like to make on this KIP when I have time (sorry
> > for the delay, Yash, and thanks for your patience!), but I did want to
> > chime in and say that I'm also not sure about overloading SinkTask::put.
> I
> > share the concerns about creating an intuitive, simple API that Yash has
> > raised. In addition, this approach doesn't seem very sustainable--what do
> > we do if we encounter another case in the future that would warrant a
> > similar solution? We probably don't want to create three, four, etc.
> > overloaded variants of the method, each of which would have to be
> > implemented by connector developers who want to both leverage the latest
> > and greatest connector APIs and maintain compatibility with connect
> > Clusters running older versions.
> >
> > I haven't been able to flesh this out into a design worth publishing in
> its
> > own KIP yet, but one alternative I've pitched to a few people with
> > generally positive interest has been to develop an official compatibility
> > library for Connect developers. This library would be released as its own
> 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Yash,

I added a sentence to the sensor() method mentioning the sensor name
must only be unique per plugin. Regarding having getters for sensors
and metrics I considered this not strictly necessary as I expect the
metrics logic in most plugins to be relatively simple. If you have a
use case that would benefit from these methods, let me know I will
reconsider.

Thanks,
Mickael


On Fri, Feb 3, 2023 at 9:16 AM Yash Mayya  wrote:
>
> Hi Mickael,
>
> Thanks for the updates.
>
> > the PluginMetrics implementation will append a
> > suffix to sensor names to unique identify
> > the plugin (based on the class name and tags).
>
> Can we call this out explicitly in the KIP, since it is important to avoid
> clashes in sensor naming? Also, should we allow plugins to retrieve sensors
> from `PluginMetrics` if we can check / verify that they own the sensor
> (based on the suffix)?
>
> Other than the above minor points, this looks good to me now!
>
> Thanks,
> Yash
>
> On Fri, Feb 3, 2023 at 2:29 AM Chris Egerton 
> wrote:
>
> > Hi Mickael,
> >
> > This is looking great. I have one small question left but I do not consider
> > it a blocker.
> >
> > What is the intended use case for PluginMetrics::close? To me at least, it
> > implies that plugin developers will be responsible for invoking that method
> > themselves in order to clean up metrics that they've created, but wouldn't
> > we want the runtime (i.e., KafkaProducer class, Connect framework, etc.) to
> > handle that automatically when the resource that the plugin applies to is
> > closed?
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Jan 26, 2023 at 10:22 AM Mickael Maison 
> > wrote:
> >
> > > Hi Yash,
> > >
> > > 1) To avoid conflicts with other sensors, the PluginMetrics
> > > implementation will append a suffix to sensor names to unique identify
> > > the plugin (based on the class name and tags). Also I changed the
> > > semantics of the sensor() method to only create sensors (originally it
> > > was get or create). If a sensor with the same name already exists, the
> > > method will throw.
> > > 2) Tags will be automatically added to metrics and sensors to unique
> > > identify the plugin. For Connect plugins, the connector name, task id
> > > and alias can be added if available. The class implementing
> > > PluginMetrics will be similar to ConnectMetrics, as in it will provide
> > > a simplified API wrapping Metrics. I'm planning to use PluginMetrics
> > > for Connect plugin too and should not need to interact with
> > > ConnectMetrics.
> > > 3) Right, I fixed the last rejected alternative.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, Jan 26, 2023 at 4:04 PM Mickael Maison  > >
> > > wrote:
> > > >
> > > > Hi Federico,
> > > >
> > > > - The metricName() method does not register anything, it just builds a
> > > > MetricName instance which is just a container holding a name, group,
> > > > description and tags for a metric. Each time it is called, it returns
> > > > a new instance. If called with the same arguments, the returned value
> > > > will be equal.
> > > > - Initially I just copied the API of Metrics. I made some small
> > > > changes so the metric and sensor methods are a bit more similar
> > > > - Good catch! I fixed the example.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > > On Thu, Jan 26, 2023 at 3:54 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > 1) I updated the KIP to only mention the interface.
> > > > > 2) This was a mistake. I've added ReplicationPolicy to the list of
> > > plugins.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, Jan 24, 2023 at 11:16 AM Yash Mayya 
> > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Thanks for the updated KIP, this is looking really good! I had a
> > > couple
> > > > > > more questions -
> > > > > >
> > > > > > 1) Sensor names need to be unique across all groups for a `Metrics`
> > > > > > instance. How are we planning to avoid naming clashes (both between
> > > > > > different plugins as well as with pre-defined sensors)?
> > > > > >
> > > > > > 2) Connect has a `ConnectMetrics` wrapper around `Metrics` via
> > which
> > > > > > rebalance / worker / connector / task metrics are recorded. Could
> > you
> > > > > > please elaborate in the KIP how the plugin metrics for connectors /
> > > tasks
> > > > > > will inter-operate with this?
> > > > > >
> > > > > > Another minor point is that the third rejected alternative appears
> > > to be an
> > > > > > incomplete sentence?
> > > > > >
> > > > > > Thanks,
> > > > > > Yash
> > > > > >
> > > > > > On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I've updated the KIP based on the feedback.
> > > > > > >
> > > > > > > Now instead of receiving a Metrics instance, plugins get access
> > to
> > > > > > > PluginMetrics 

Re: [DISCUSS] KIP-898: Modernize Connect plugin discovery

[Chris Egerton]

Hi Greg,

Thanks for the updates. Unless stated below, I agree with your responses. A
few more thoughts:

1. IMO it's not worth it to have separate commands for adding/removing
manifests, mostly because it adds complexity to the tool and might make it
harder for users to understand. I think a single "update-manifests" or
"sync-manifests" command that both adds manifests for found plugins and
removes manifests for unfound plugins by default, with a --keep-unfound
flag to disable removal on an opt-in basis, would make more sense.

2. I agree with this point you raise about deprecation: "if we agree that
the old mechanism has some sunset date that we just don't know yet, we
should still communicate to users that the sunset date is somewhere in the
future." However, I'm not certain that releasing the features proposed in
this KIP alone gives us enough confidence in them to sunset the legacy
plugin loading logic, which is why I was hoping we could have at least one
successful release (preferably even two or three) with this feature in
order to identify potential gaps in the design before proceeding. This is
similar to the strategy we took with incremental rebalancing in Connect,
which introduced the "connect.protocol" property with the proposal that it
could be marked deprecated in the next major release. That strategy served
us well, since there were a few kinks in the logic for that feature that
needed to be worked out, and many users continued to rely on the legacy
rebalancing logic as a workaround for those issues.

3. I was wondering about error handling with the service loading mechanism
because the Javadocs for the Iterator returned by ServiceLoader::iterator
[1] state that "If an error is thrown then subsequent invocations of the
iterator will make a best effort to locate and instantiate the next
available provider, but in general such recovery cannot be guaranteed."
Upon further reflection, I agree that it's fine for now to leave details on
error handling out of the public contract for this new loading mode, but
only if we also agree to hold off on deprecating the old loading mode,
since there may be cases where worker startup is crippled by an
improperly-packaged plugin when using the service loader mechanism for its
plugins.

[1] -
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/ServiceLoader.html#iterator()

Cheers,

Chris

On Tue, Jan 24, 2023 at 10:26 AM Federico Valeri 
wrote:

> Hi Greg,
>
> On Tue, Jan 24, 2023 at 2:48 AM Greg Harris
>  wrote:
> >
> > Federico,
> >
> > Thanks for taking a look!
> >
> > > I was just wondering if we should use a better script name like
> > > "connect-convert-to-service-provider.sh" or something like this
> >
> > I agree that the current name is not ideal, while the current name
> > describes part of what it is doing, it does not describe all of what it
> is
> > doing, or why that is important to the caller.
> > I looked around to see what style the other Kafka commands use, and it
> > seems like there's not a standard pattern. Some of the scripts are
> > noun-phrases, and some are verb-phrases.
> > Some take commands like `create`, `format`, etc, and some take options
> like
> > `--generate`, `--execute`, `--verify`.
> > I've updated the command to `bin/connect-plugin-path.sh
> > (list|add-manifests|remove-manifests) --worker-config `,
> let
> > me know if that's better or worse than `bin/connect-scan-plugin-path.sh`.
> >
> > > maybe add a --dry-run option.
> >
> > Thanks for the suggestion, I've updated the KIP to include a --dry-run
> > option with some typical semantics.
>
> This is much better. Thanks!
>
> I think it's better to not deprecate the add-manifests and
> remove-manifests script sub-commands. When we will remove the
> deprecated plugin discovery modes, one may still have the need to
> convert an old connector release, maybe because it's the only version
> compatible with a third-party external system.
>
> >
> > Chris,
> >
> > I'm glad you liked the personas! Since this KIP requires others' help to
> > improve the ecosystem, I needed a way to make sure everyone knows what
> role
> > they have to play. I hope I've accomplished that.
> >
> > > 1. Will workers' resiliency to packaging issues be affected when the
> new
> > > scanning behavior is enabled? For example, will a single
> poorly-packaged
> > > plugin prevent other correctly-packaged plugins from being loaded, or
> > > worse, cause the worker to fail? And either way, are there any details
> > that
> > > we can call out to back up any guarantees we want to provide on that
> > front?
> >
> > The current behavior when encountering packaging issues seems to be to
> log
> > errors and continue, and that will certainly continue for the ONLY_SCAN
> and
> > HYBID_WARN modes.
> > As written, the HYBRID_FAIL mode breaks from this trend and will throw
> > exceptions that crash the worker only when the connectors are missing
> from
> > serviceloader discovery.
> > The behavior for 

Re: [VOTE] 3.4.0 RC2

[Ismael Juma]

Thanks David and Sophie!

Ismael

On Mon, Feb 6, 2023, 7:22 AM David Arthur  wrote:

> I'd like to go ahead and close out this vote. There were three binding
> votes and one non-binding vote.
>
> Binding +1 PMC votes:
> * David Jacot
> * Bill Bejeck
> * Mickael Maison
>
> Non-binding community votes:
> * Federico Valeri
>
> There were no -1 votes.
>
> The vote for Apache Kafka 3.4.0 RC2 passes.
>
> Thanks to all who voted and a big thanks to Sophie who did a vast majority
> of the work to prepare this release. We will continue the release process
> and get the announcement sent out this week.
>
> Cheers,
> David Arthur
>
> On Fri, Feb 3, 2023 at 12:25 PM Federico Valeri 
> wrote:
>
> > +1 (non binding)
> >
> > - Ran the unit and integration test suites with Java 17 and Scala 2.13
> > - Ran a series of basic examples and client configurations
> > - Spot checked the docs and Javadocs
> >
> > Thanks
> > Fede
> >
> > On Fri, Feb 3, 2023 at 5:29 PM Jakub Scholz  wrote:
> > >
> > > +1 (non-binding). I run my tests with the staged Scala 2.13 binaries
> and
> > > staged Maven artifacts. All seems to work fine.
> > >
> > > Thanks & Regards
> > > Jakub
> > >
> > > On Tue, Jan 31, 2023 at 8:01 PM David Arthur 
> > wrote:
> > >
> > > > Hey folks, we found a couple of blockers with RC1 and have fixed them
> > in
> > > > the latest release candidate, RC2.
> > > >
> > > > The major features of this release include:
> > > >
> > > > * KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> > > > >
> > > >
> > > > * KIP-876: Time based cluster metadata snapshots
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-876%3A+Time+based+cluster+metadata+snapshots
> > > > >
> > > >
> > > > * KIP-787: MM2 manage Kafka resources with custom Admin
> implementation.
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335620
> > > > >
> > > >
> > > > * KIP-866 ZooKeeper to KRaft Migration
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
> > > > >
> > > > (Early
> > > > Access)
> > > >
> > > >
> > > >
> > > > Release notes for the 3.4.0 release:
> > > >
> > > >
> > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/RELEASE_NOTES.html
> > > >
> > > >
> > > > Please download, test and vote by Friday, February 3, 5pm PT
> > > >
> > > >
> > > > ---
> > > >
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >
> > > > https://kafka.apache.org/KEYS
> > > >
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > >
> > > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/
> > > >
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > >
> > > > * Javadoc:
> > > >
> > > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/javadoc/
> > > >
> > > >
> > > > * Tag to be voted upon (off 3.4 branch) is the 3.4.0 tag:
> > > >
> > > > https://github.com/apache/kafka/releases/tag/3.4.0-rc2
> > > >
> > > >
> > > > * Documentation:
> > > >
> > > > https://kafka.apache.org/34/documentation.html
> > > >
> > > >
> > > > * Protocol:
> > > >
> > > > https://kafka.apache.org/34/protocol.html
> > > >
> > > >
> > > > ---
> > > >
> > > >
> > > > Test results:
> > > >
> > > >
> > > > We haven't had a 100% passing build, but the latest system test run
> > looks
> > > > pretty good:
> > > >
> > > >
> >
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.4/2023-01-31--001.system-test-kafka-3.4--1675184554--confluentinc--3.4--ef3f5bd834/report.html
> > > >
> > > >
> > > > Here are the Jenkins test runs for 3.4:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/. We will
> > > > continue
> > > > trying to diagnose the flaky test failures as the release continues.
> I
> > do
> > > > not expect that any of these test failures are blockers for the
> > release.
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > David Arthur
> > > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1559

[Apache Jenkins Server]

See 

Re: [VOTE] 3.4.0 RC2

[David Arthur]

I'd like to go ahead and close out this vote. There were three binding
votes and one non-binding vote.

Binding +1 PMC votes:
* David Jacot
* Bill Bejeck
* Mickael Maison

Non-binding community votes:
* Federico Valeri

There were no -1 votes.

The vote for Apache Kafka 3.4.0 RC2 passes.

Thanks to all who voted and a big thanks to Sophie who did a vast majority
of the work to prepare this release. We will continue the release process
and get the announcement sent out this week.

Cheers,
David Arthur

On Fri, Feb 3, 2023 at 12:25 PM Federico Valeri 
wrote:

> +1 (non binding)
>
> - Ran the unit and integration test suites with Java 17 and Scala 2.13
> - Ran a series of basic examples and client configurations
> - Spot checked the docs and Javadocs
>
> Thanks
> Fede
>
> On Fri, Feb 3, 2023 at 5:29 PM Jakub Scholz  wrote:
> >
> > +1 (non-binding). I run my tests with the staged Scala 2.13 binaries and
> > staged Maven artifacts. All seems to work fine.
> >
> > Thanks & Regards
> > Jakub
> >
> > On Tue, Jan 31, 2023 at 8:01 PM David Arthur 
> wrote:
> >
> > > Hey folks, we found a couple of blockers with RC1 and have fixed them
> in
> > > the latest release candidate, RC2.
> > >
> > > The major features of this release include:
> > >
> > > * KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> > > >
> > >
> > > * KIP-876: Time based cluster metadata snapshots
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-876%3A+Time+based+cluster+metadata+snapshots
> > > >
> > >
> > > * KIP-787: MM2 manage Kafka resources with custom Admin implementation.
> > > <
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335620
> > > >
> > >
> > > * KIP-866 ZooKeeper to KRaft Migration
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
> > > >
> > > (Early
> > > Access)
> > >
> > >
> > >
> > > Release notes for the 3.4.0 release:
> > >
> > >
> https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/RELEASE_NOTES.html
> > >
> > >
> > > Please download, test and vote by Friday, February 3, 5pm PT
> > >
> > >
> > > ---
> > >
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >
> > > https://kafka.apache.org/KEYS
> > >
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > >
> > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/
> > >
> > >
> > > * Maven artifacts to be voted upon:
> > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > >
> > > * Javadoc:
> > >
> > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/javadoc/
> > >
> > >
> > > * Tag to be voted upon (off 3.4 branch) is the 3.4.0 tag:
> > >
> > > https://github.com/apache/kafka/releases/tag/3.4.0-rc2
> > >
> > >
> > > * Documentation:
> > >
> > > https://kafka.apache.org/34/documentation.html
> > >
> > >
> > > * Protocol:
> > >
> > > https://kafka.apache.org/34/protocol.html
> > >
> > >
> > > ---
> > >
> > >
> > > Test results:
> > >
> > >
> > > We haven't had a 100% passing build, but the latest system test run
> looks
> > > pretty good:
> > >
> > >
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.4/2023-01-31--001.system-test-kafka-3.4--1675184554--confluentinc--3.4--ef3f5bd834/report.html
> > >
> > >
> > > Here are the Jenkins test runs for 3.4:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/. We will
> > > continue
> > > trying to diagnose the flaky test failures as the release continues. I
> do
> > > not expect that any of these test failures are blockers for the
> release.
> > >
> > >
> > > Thanks!
> > >
> > > David Arthur
> > >
>

[jira] [Created] (KAFKA-14679) Add new __consumer_offsets records

[David Jacot (Jira)]

David Jacot created KAFKA-14679:
---

 Summary: Add new __consumer_offsets records
 Key: KAFKA-14679
 URL: https://issues.apache.org/jira/browse/KAFKA-14679
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14678) Move __consumer_offsets records from core to group-coordinator

[David Jacot (Jira)]

David Jacot created KAFKA-14678:
---

 Summary: Move __consumer_offsets records from core to 
group-coordinator
 Key: KAFKA-14678
 URL: https://issues.apache.org/jira/browse/KAFKA-14678
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add "Security Implications" section to KIP template

[Chris Egerton]

Hi Bruno,

Apologies for the delay; covid paid me a visit recently and took me away
from this discussion for a bit.

I'm fine with adding specific examples to help flesh out the questions we
want to think about while writing/reviewing KIPs, but IMO it's important to
emphasize that they are not comprehensive and that it's not enough to
address each example individually instead of considering the high-level
questions that they apply to.

RE creating/configuring Kafka clients with arbitrary settings: I was
imagining creating/configuring them on a machine that you do not otherwise
have access to, a detail that should definitely be added to the template if
we still want to call out this scenario. Creating/configuring a Kafka
client on a machine that you do not own can be used to, e.g., perform a DOS
attack on an arbitrary hostname by instantiating a large number of clients
that use that hostname for their bootstrap server.

Taking recent discussion into account, here's a revised draft for the
proposed "Security Implications" section:


How does this impact the security of the project?

Does it make Kafka or any of its components (brokers, clients, Kafka
Connect, Kafka Streams, Mirror Maker 2, etc.) less secure when run with
default settings?
• Does it give access to the file system or execute code on the machine
running Kafka or one of its components?
• Does it allow users to create or arbitrarily configure Kafka clients on a
machine that they otherwise do not have access to?
• Does it give unauthenticated users new ways to access topics, connectors,
or other Kafka-related resources?
• Does it allow users to allocate an unbounded amount of some resource on
the user's machine, such as heap memory, network connections, or disk space?

Does it make Kafka or any of its components more difficult to run in a
fully-secured fashion?
• Does it introduce a new API such as a REST server that, if left
unsecured, could be used by malicious users to attack the component or the
machine it's running on?
• Does it require additional Kafka ACLs for the component to run
successfully, such as permission to access a new transactional producer ID?
• Does it require new access to resources on the user's machine, such as
permission to create, read, or modify files?
• Does it require new access to resources in the user's environment, such
as permission to issue arbitrary network requests?


Looking forward to your thoughts!

Cheers,

Chris

On Mon, Jan 16, 2023 at 4:47 AM Bruno Cadonna  wrote:

> Hi Chris,
>
> Thanks for the clarifications!
>
> I am in favor to formulate the questions more concretely, like:
>
>
> Does the KIP access the file system or execute code on the machine
> running Kafka or one of its components?
>
> Does the KIP introduce new components that need to be secured separately?
>
>
> Probably, this list needs to be extended.
>
> I think, we need to be as concrete as possible to get the info we want
> in a KIP to be able to review it well.
>
>
>  From the examples you provided, I do not understand the following. What
> does it mean to "create or configure Kafka clients with arbitrary
> settings"? Independently of a change in a KIP, one can always create or
> configure a Kafka client with arbitrary settings. The brokers need to
> take care of such cases anyways. I do not see how this extends the
> attack surface. But maybe I am missing something here.
>
>
> Best,
> Bruno
>
>
> On 12.01.23 20:05, Chris Egerton wrote:
> > Hi Luke and Bruno,
> >
> > Thanks for taking a look! Happy to provide some examples here to clarify
> > the points, and if they seem useful enough, we can also add them to the
> > template.
> >
> >> Does it make Kafka or any of its components (brokers, clients, Kafka
> > Connect, Kafka Streams, Mirror Maker 2, etc.) less secure when run with
> > default settings?
> >
> > Examples include allowing unauthenticated users to access the file system
> > of, or execute code on, the machine running Kafka/one of its components,
> or
> > create or configure Kafka clients with arbitrary settings
> >
> >> Does it give users new access to configure clients, brokers, topics,
> etc.
> > in situations where they did not have this access before? Keep in mind
> that
> > the ability to arbitrarily configure a Kafka client can add to the attack
> > surface of a project and may be safer to disable by default.
> >
> > With examples provided, this point is likely made redundant by the
> > first/third points
> >
> >> Does it make Kafka or any of its components more difficult to run in a
> > fully-secured fashion?
> >
> > Examples include requiring new ACLs to run existing components (e.g.,
> > requiring write permission for a specific transactional ID in order to
> > start Kafka Connect), or adding new APIs that, if left unsecured, would
> > leave the component vulnerable to malicious users (e.g., adding a REST
> > server to Kafka Streams that allows topologies to be dynamically
> > manipulated).
> >
> > I hope this helps; 

[DISCUSS] KIP-904 Kafka Streams - Guarantee subtractor is called before adder if key has not changed

[Fq Public]

Hi everyone,

I'd like to share a new KIP for discussion:
https://cwiki.apache.org/confluence/x/P5VbDg

This could be considered mostly as a "bug fix" but we wanted to raise a KIP
for discussion because it involves changes to the serialization format of
an internal topic which raises backward compatibility considerations.

Please take a look and let me know what you think.

Thanks,
Farooq

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1558

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14677) Dependency conversion error

[Jan Hendriks (Jira)]

Jan Hendriks created KAFKA-14677:


 Summary: Dependency conversion error
 Key: KAFKA-14677
 URL: https://issues.apache.org/jira/browse/KAFKA-14677
 Project: Kafka
  Issue Type: Bug
  Components: build
Affects Versions: 3.1.2
 Environment: Maven 3.8.4, Java OpenJDK 11 & 17 & 19, Ubuntu Linux 
22.04.01 LTS
Reporter: Jan Hendriks
 Attachments: pom.xml

Compilation of projects using spring-kafka with Spring-Boot 2.7.7 fails if 
maven-enforcer-plugin is enabled with dependency conversion, as there are 
multiple mismatching dependency versions being dragged in, see below:

Compilation failures are as follows (potentially incomplete):
{code:java}
mvn clean install
[...]
[INFO] — maven-enforcer-plugin:3.0.0-M3:enforce (enforce) @ 
SpringKafkaDependencyIssue —
[WARNING]
Dependency convergence error for org.scala-lang:scala-library:2.13.6 paths to 
dependency are:
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang:scala-library:2.13.6
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-com.fasterxml.jackson.module:jackson-module-scala_2.13:2.13.4
+-org.scala-lang:scala-library:2.13.6
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang.modules:scala-collection-compat_2.13:2.4.4
+-org.scala-lang:scala-library:2.13.5
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang.modules:scala-java8-compat_2.13:1.0.0
+-org.scala-lang:scala-library:2.13.5
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang:scala-reflect:2.13.6
+-org.scala-lang:scala-library:2.13.6
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-com.typesafe.scala-logging:scala-logging_2.13:3.9.3
+-org.scala-lang:scala-library:2.13.4
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang:scala-library:2.13.6
[WARNING]
Dependency convergence error for org.scala-lang:scala-reflect:2.13.6 paths to 
dependency are:
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang:scala-reflect:2.13.6
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-com.typesafe.scala-logging:scala-logging_2.13:3.9.3
+-org.scala-lang:scala-reflect:2.13.4
and
+-org.example:SpringKafkaDependencyIssue:1.0-SNAPSHOT
+-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
+-org.scala-lang:scala-reflect:2.13.6
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
failed with message:
Failed while enforcing releasability. See above detailed error message.
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1.170 s
[INFO] Finished at: 2023-01-30T13:21:27+01:00
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce (enforce) on 
project SpringKafkaDependencyIssue: Some Enforcer rules have failed. Look above 
for specific messages explaining why the rule failed. -> [Help 1]
[...] {code}
and, excerpt from another project:
{code:java}
[...]
Dependency convergence error for org.apache.kafka:kafka-metadata:3.1.2 paths to 
dependency are:
+-org.example:SpringKafkaDependencyIssue:4.39.3-SNAPSHOT
  +-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka-metadata:3.1.2
and
+-org.example:SpringKafkaDependencyIssue:4.39.3-SNAPSHOT
  +-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka-metadata:3.1.2
  +-org.apache.kafka:kafka-metadata:2.8.2
and
+-org.example:SpringKafkaDependencyIssue:4.39.3-SNAPSHOT
  +-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
  +-org.apache.kafka:kafka-metadata:3.1.2
and
+-org.example:SpringKafkaDependencyIssue:4.39.3-SNAPSHOT
  +-org.springframework.kafka:spring-kafka-test:2.8.11
+-org.apache.kafka:kafka_2.13:3.1.2
  +-org.apache.kafka:kafka-metadata:3.1.2
[...] {code}
 

Reproducer 

[jira] [Created] (KAFKA-14676) Token endpoint URL used for OIDC cannot be set on the JAAS config

[Rajini Sivaram (Jira)]

Rajini Sivaram created KAFKA-14676:
--

 Summary: Token endpoint URL used for OIDC cannot be set on the 
JAAS config
 Key: KAFKA-14676
 URL: https://issues.apache.org/jira/browse/KAFKA-14676
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 3.3.2, 3.2.3, 3.1.2, 3.4.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


Kafka allows multiple clients within a JVM to use different SASL configurations 
by configuring the JAAS configuration in `sasl.jaas.config` instead of the 
JVM-wide system property. For SASL login, we reuse logins within a JVM by 
caching logins indexed by their sasl.jaas.config. This relies on login configs 
being overridable using `sasl.jaas.config`. 

KIP-768 
([https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575)] 
added support for OIDC for SASL/OAUTHBEARER. The token endpoint used to acquire 
tokens can currently only be configured using the Kafka config 
`sasl.oauthbearer.token.endpoint.url`. This prevents different clients within a 
JVM from using different URLs. We need to either provide a way to override the 
URL within `sasl.jaas.config` or include more of the client configs in the 
LoginMetadata used as key for cached logins.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi!

There seems to be not much more discussion going, so I'm planning to start
the vote in a couple of days.

Thanks,

Ivan

On Wed, 18 Jan 2023 at 12:06, Ivan Yurchenko 
wrote:

> Hello!
> I would like to start the discussion thread on KIP-899: Allow clients to
> rebootstrap.
> This KIP proposes to allow Kafka clients to repeat the bootstrap process
> when fetching metadata if none of the known nodes are available.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
>
> A question right away: should we eventually change the default behavior or
> it can remain configurable "forever"? The latter is proposed in the KIP.
>
> Thank you!
>
> Ivan
>
>
>