Re: [VOTE] KIP-890: Transactions Server Side Defense

[David Jacot]

Thanks for the KIP, Justine. +1 (binding)

On Fri, Feb 3, 2023 at 1:36 AM Matthias J. Sax  wrote:

> Thanks for the KIP!
>
> +1 (binding)
>
>
> On 2/2/23 4:18 PM, Artem Livshits wrote:
> > (non-binding) +1.  Looking forward to the implementation and fixing the
> > issues that we've got.
> >
> > -Artem
> >
> > On Mon, Jan 23, 2023 at 2:25 PM Guozhang Wang <
> guozhang.wang...@gmail.com>
> > wrote:
> >
> >> Thanks Justine, I have no further comments on the KIP. +1.
> >>
> >> On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson
> >>  wrote:
> >>>
> >>> +1. Thanks Justine!
> >>>
> >>> -Jason
> >>>
> >>> On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy 
> >> wrote:
> >>>
>  (non-binding) +1. Thank you for the KIP, Justine! I've read it; it
> >> makes
>  sense to me and I am excited for the implementation.
> 
>  Colt McNealy
>  *Founder, LittleHorse.io*
> 
> 
>  On Tue, Jan 10, 2023 at 10:46 AM Justine Olshan
>   wrote:
> 
> > Hi everyone,
> >
> > I would like to start a vote on KIP-890 which aims to prevent some
> >> of the
> > common causes of hanging transactions and make other general
> >> improvements
> > to transactions in Kafka.
> >
> >
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
> >
> > Please take a look if you haven't already and vote!
> >
> > Justine
> >
> 
> >>
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1554

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-890: Transactions Server Side Defense

[Matthias J. Sax]

Thanks for the KIP!

+1 (binding)


On 2/2/23 4:18 PM, Artem Livshits wrote:

(non-binding) +1.  Looking forward to the implementation and fixing the
issues that we've got.

-Artem

On Mon, Jan 23, 2023 at 2:25 PM Guozhang Wang 
wrote:


Thanks Justine, I have no further comments on the KIP. +1.

On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson
 wrote:


+1. Thanks Justine!

-Jason

On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy 

wrote:



(non-binding) +1. Thank you for the KIP, Justine! I've read it; it

makes

sense to me and I am excited for the implementation.

Colt McNealy
*Founder, LittleHorse.io*


On Tue, Jan 10, 2023 at 10:46 AM Justine Olshan
 wrote:


Hi everyone,

I would like to start a vote on KIP-890 which aims to prevent some

of the

common causes of hanging transactions and make other general

improvements

to transactions in Kafka.






https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense


Please take a look if you haven't already and vote!

Justine

Re: [VOTE] KIP-890: Transactions Server Side Defense

[Artem Livshits]

(non-binding) +1.  Looking forward to the implementation and fixing the
issues that we've got.

-Artem

On Mon, Jan 23, 2023 at 2:25 PM Guozhang Wang 
wrote:

> Thanks Justine, I have no further comments on the KIP. +1.
>
> On Tue, Jan 17, 2023 at 10:34 AM Jason Gustafson
>  wrote:
> >
> > +1. Thanks Justine!
> >
> > -Jason
> >
> > On Tue, Jan 10, 2023 at 3:46 PM Colt McNealy 
> wrote:
> >
> > > (non-binding) +1. Thank you for the KIP, Justine! I've read it; it
> makes
> > > sense to me and I am excited for the implementation.
> > >
> > > Colt McNealy
> > > *Founder, LittleHorse.io*
> > >
> > >
> > > On Tue, Jan 10, 2023 at 10:46 AM Justine Olshan
> > >  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I would like to start a vote on KIP-890 which aims to prevent some
> of the
> > > > common causes of hanging transactions and make other general
> improvements
> > > > to transactions in Kafka.
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense
> > > >
> > > > Please take a look if you haven't already and vote!
> > > >
> > > > Justine
> > > >
> > >
>

[jira] [Created] (KAFKA-14675) Extract metadata-related tasks from Fetcher into MetadataFetcher

[Kirk True (Jira)]

Kirk True created KAFKA-14675:
-

 Summary: Extract metadata-related tasks from Fetcher into 
MetadataFetcher
 Key: KAFKA-14675
 URL: https://issues.apache.org/jira/browse/KAFKA-14675
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


Extract from {{Fetcher}} the APIs that are related to metadata operations into 
a new class named {{{}MetadataFetcher{}}}. This will allow the refactoring of 
{{Fetcher}} and {{MetadataFetcher}} for the new consumer. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14660) Divide by zero security vulnerability (sonatype-2019-0422)

[Andy Coates (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Coates reopened KAFKA-14660:
-

The issue here is more the SonaType security vulnerability report than any 
impossible to reach divide by zero issue. Unfortunately, I'm struggling to find 
information on _how_ to mark the vulnerability resolved in SonaType.  This was 
why I was suggesting opening and merging the PR, as it seems the PR is the 
cause of the report.

I realise the PR's solution wasn't ideal. Hence I was suggesting to merge and 
put in a second change after to fix the fix, so to speak.

If you've already summited a fix for the DBZ, then I see two potential ways 
forward:
 # work out how to inform SonaType the issue is fixed:
 ## There is a [Report 
correction|https://ossindex.sonatype.org/doc/report-vulnerability] link on the 
bug report.  May you, or I if you let me know the PR you fixed the DBZ in, can 
use this to raise the fact its been fixed?
 ## Maybe just tagging the [SonaType 
issue|https://ossindex.sonatype.org/vulnerability/sonatype-2019-0422?component-type=maven=org.apache.kafka%2Fkafka-streams_source=ossindex-client_medium=integration_content=1.7.0]
 in your PR would be enough?
 ## Does someone in Confluent know about this stuff that you can talk to?
 ## 
 # reopen, 'adjust' and merge the original PR... hopefully triggering SonaType 
to mark the issue resolved.

> Divide by zero security vulnerability (sonatype-2019-0422)
> --
>
> Key: KAFKA-14660
> URL: https://issues.apache.org/jira/browse/KAFKA-14660
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.3.2
>Reporter: Andy Coates
>Assignee: Matthias J. Sax
>Priority: Minor
> Fix For: 3.5.0
>
>
> Looks like SonaType has picked up a "Divide by Zero" issue reported in a PR 
> and, because the PR was never merged, is now reporting it as a security 
> vulnerability in the latest Kafka Streams library.
>  
> See:
>  * [Vulnerability: 
> sonatype-2019-0422]([https://ossindex.sonatype.org/vulnerability/sonatype-2019-0422?component-type=maven=org.apache.kafka%2Fkafka-streams_source=ossindex-client_medium=integration_content=1.7.0)]
>  * [Original PR]([https://github.com/apache/kafka/pull/7414])
>  
> While it looks from the comments made by [~mjsax] and [~bbejeck] that the 
> divide-by-zero is not really an issue, the fact that its now being reported 
> as a vulnerability is, especially with regulators.
> PITA, but we should consider either getting this vulnerability removed 
> (Google wasn't very helpful in providing info on how to do this), or fixed 
> (Again, not sure how to tag the fix as fixing this issue).  One option may 
> just be to reopen the PR and merge (and then fix forward by switching it to 
> throw an exception).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Chris Egerton]

Hi Mickael,

This is looking great. I have one small question left but I do not consider
it a blocker.

What is the intended use case for PluginMetrics::close? To me at least, it
implies that plugin developers will be responsible for invoking that method
themselves in order to clean up metrics that they've created, but wouldn't
we want the runtime (i.e., KafkaProducer class, Connect framework, etc.) to
handle that automatically when the resource that the plugin applies to is
closed?

Cheers,

Chris

On Thu, Jan 26, 2023 at 10:22 AM Mickael Maison 
wrote:

> Hi Yash,
>
> 1) To avoid conflicts with other sensors, the PluginMetrics
> implementation will append a suffix to sensor names to unique identify
> the plugin (based on the class name and tags). Also I changed the
> semantics of the sensor() method to only create sensors (originally it
> was get or create). If a sensor with the same name already exists, the
> method will throw.
> 2) Tags will be automatically added to metrics and sensors to unique
> identify the plugin. For Connect plugins, the connector name, task id
> and alias can be added if available. The class implementing
> PluginMetrics will be similar to ConnectMetrics, as in it will provide
> a simplified API wrapping Metrics. I'm planning to use PluginMetrics
> for Connect plugin too and should not need to interact with
> ConnectMetrics.
> 3) Right, I fixed the last rejected alternative.
>
> Thanks,
> Mickael
>
> On Thu, Jan 26, 2023 at 4:04 PM Mickael Maison 
> wrote:
> >
> > Hi Federico,
> >
> > - The metricName() method does not register anything, it just builds a
> > MetricName instance which is just a container holding a name, group,
> > description and tags for a metric. Each time it is called, it returns
> > a new instance. If called with the same arguments, the returned value
> > will be equal.
> > - Initially I just copied the API of Metrics. I made some small
> > changes so the metric and sensor methods are a bit more similar
> > - Good catch! I fixed the example.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Thu, Jan 26, 2023 at 3:54 PM Mickael Maison 
> wrote:
> > >
> > > Hi Chris,
> > >
> > > 1) I updated the KIP to only mention the interface.
> > > 2) This was a mistake. I've added ReplicationPolicy to the list of
> plugins.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Jan 24, 2023 at 11:16 AM Yash Mayya 
> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the updated KIP, this is looking really good! I had a
> couple
> > > > more questions -
> > > >
> > > > 1) Sensor names need to be unique across all groups for a `Metrics`
> > > > instance. How are we planning to avoid naming clashes (both between
> > > > different plugins as well as with pre-defined sensors)?
> > > >
> > > > 2) Connect has a `ConnectMetrics` wrapper around `Metrics` via which
> > > > rebalance / worker / connector / task metrics are recorded. Could you
> > > > please elaborate in the KIP how the plugin metrics for connectors /
> tasks
> > > > will inter-operate with this?
> > > >
> > > > Another minor point is that the third rejected alternative appears
> to be an
> > > > incomplete sentence?
> > > >
> > > > Thanks,
> > > > Yash
> > > >
> > > > On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've updated the KIP based on the feedback.
> > > > >
> > > > > Now instead of receiving a Metrics instance, plugins get access to
> > > > > PluginMetrics that exposes a much smaller API. I've removed the
> > > > > special handling for connectors and tasks and they must now
> implement
> > > > > the Monitorable interface as well to use this feature. Finally the
> > > > > goal is to allow all plugins (apart from metrics reporters) to use
> > > > > this feature. I've listed them all (there are over 30 pluggable
> APIs)
> > > > > but I've not added the list in the KIP. The reason is that new
> plugins
> > > > > could be added in the future and instead I'll focus on adding
> support
> > > > > in all the place that instantiate classes.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Chris/Yash,
> > > > > >
> > > > > > Thanks for taking a look and providing feedback.
> > > > > >
> > > > > > 1) Yes you're right, when using incompatible version, metrics()
> would
> > > > > > trigger NoSuchMethodError. I thought using the context to pass
> the
> > > > > > Metrics object would be more idiomatic for Connect but maybe
> > > > > > implementing Monitorable would be simpler. It would also allow
> other
> > > > > > Connect plugins (transformations, converters, etc) to register
> > > > > > metrics. So I'll make that change.
> > > > > >
> > > > > > 2) As mentioned in the rejected alternatives, I considered
> having a
> > > > > > PluginMetrics class/interface with a limited API. But since
> Metrics 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1553

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 442387 lines...]
[2023-02-02T19:11:59.068Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-02-02T19:12:00.189Z] 
[2023-02-02T19:12:00.189Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-02-02T19:12:00.189Z] 
[2023-02-02T19:12:00.189Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-02-02T19:12:00.189Z] 
[2023-02-02T19:12:00.190Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-02-02T19:12:00.190Z] 
[2023-02-02T19:12:00.190Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-02-02T19:12:00.190Z] 
[2023-02-02T19:12:00.190Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-02-02T19:12:00.190Z] 
[2023-02-02T19:12:00.190Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-02-02T19:12:01.135Z] 
[2023-02-02T19:12:01.135Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-02-02T19:12:01.135Z] 
[2023-02-02T19:12:01.135Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testDeletePath() STARTED
[2023-02-02T19:12:01.135Z] 
[2023-02-02T19:12:01.135Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testDeletePath() PASSED
[2023-02-02T19:12:01.135Z] 
[2023-02-02T19:12:01.135Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateTokenChangeNotification() STARTED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateTokenChangeNotification() PASSED
[2023-02-02T19:12:02.177Z] 
[2023-02-02T19:12:02.177Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetTopicsAndPartitions() STARTED
[2023-02-02T19:12:03.145Z] 
[2023-02-02T19:12:03.145Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 177 > DynamicBrokerReconfigurationTest > 
testAdvertisedListenerUpdate() STARTED
[2023-02-02T19:12:03.318Z] 
[2023-02-02T19:12:03.318Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetTopicsAndPartitions() PASSED
[2023-02-02T19:12:03.318Z] 
[2023-02-02T19:12:03.318Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED
[2023-02-02T19:12:03.318Z] 
[2023-02-02T19:12:03.318Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED
[2023-02-02T19:12:03.318Z] 
[2023-02-02T19:12:03.318Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED
[2023-02-02T19:12:04.436Z] 
[2023-02-02T19:12:04.436Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED
[2023-02-02T19:12:04.436Z] 
[2023-02-02T19:12:04.436Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testRegisterBrokerInfo() STARTED
[2023-02-02T19:12:04.436Z] 
[2023-02-02T19:12:04.436Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testRegisterBrokerInfo() PASSED
[2023-02-02T19:12:04.436Z] 
[2023-02-02T19:12:04.436Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testRetryRegisterBrokerInfo() STARTED

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

[Omnia Ibrahim]

Hi Luke and Justine,
Are there any thoughts or updates on this? I would love to help with this
as we are hitting this more frequently now.

best,

On Mon, Oct 31, 2022 at 6:15 PM Omnia Ibrahim 
wrote:

> Hi Luke and Justine,
>
>> For (3), you said:
>> > - I have some concerns about the impact of this option on the
>> transactional
>> producers, for example, what will happen to an ongoing transaction
>> associated with an expired PID? Would this leave the transactions in a
>> "hanging" state?
>>
>> - How will we notify the client that the transaction can't continue due to
>> an expired PID?
>>
>> - If PID got marked as `expired` this will mean that
>> `admin.DescribeProducers` will not list them which will make
>> *`kafka-transactions.sh
>> --list`* a bit tricky as we can't identify if there are transactions
>> linked
>> to this expired PID or not. The same concern applies to
>> *`kafka-transactions.sh
>> --find-hanging`*.
>>
>> --> Yes, you're right. Those are also concerns for this solution.
>> Currently, there's no way to notify clients about the expiration.
>> Also, the ongoing transactions will be hanging. For the admin cli, we've
>> never thought about it. Good point.
>> In summary, to adopt this solution, there are many issues needed to get
>> fixed.
>>
>
> Justin already clarified that if PID is attached to a transaction it will
> not expire so identifying the transactions shouldn't be a concern anymore.
> The only concern here will be that this solution will not solve the
> problem if the rouge producer is a transactional producer with hanging
> transactions.
> If anyone faced this situation they will need to abort the hanging
> transactions manually and then the solution to expire a PID can then work.
>
> --> Yes, I mean KafkaPrinciple (sorry, I didn't make it clear)
>> Yes, We were thinking about throttling by KafkaPrinciple. Client Id is
>> also
>> workable.
>> It's just these 2 attributes are not required.
>> That is, it's possible we take all clients as the same one: {default
>> KafkaPrinciple + default clientID}, and apply throttling on it.
>> Do you have any thoughts about it?
>> Maybe skip throttling for {default KafkaPrinciple + default clientID}
>>
>
> Throttling for default KafkaPrinciple and default ClientID is useful when
> we need to have a hard limit on the whole cluster and whoever is running
> the cluster doesn't knowclientsntIDs or if a KafkaPrinciple is reused
> between different producer applications.
> I usually find it helpful to have a way to apply throttling only on the
> rough clients only once I identify them without punishing everyone on the
> cluster. However, there are two problems with this
> - There's no easy way at the moment to link PIDs to clientId or
> KafkaPrinciple. This need to be addressed first.
> - Is Justin's comment on the throttling, and the fact that will mean we
> either block all requests or have to store the request in memory which in
> both cases has side downs on the producer experince.
>
> I recently had another discussion with my team and it does seem like there
>> should be a way to make it more clear to the clients what is going on. A
>> lot of this protocol is implicit. I'm wondering if maybe there is a way to
>> improve the story for newer clients. (Ie if we choose to expire based on a
>> size limit, we should include a response indicating the ID has expired.)
>> We
>> also discussed ways to redefine the guarantees so that users who have
>> stronger idempotency requirements can ensure them (over
>> availability/memory
>> concerns). Let me know if you have any ideas here.
>>
>
> It may be easier to improve the experience for new clients. However, if we
> improved only the new clients we may need a way to help teams who run Kafka
> with rough clients on old versions by at least giving them an easy way to
> identify the clientId/ or KafkaPrinciple that generated these PIDs.
>
> For context, it's very tricky to even identify which clientId is creating
> all these PIDs that caused OOM, which is a contributing part of the issue
> at the moment. So maybe one option here could be adding a new metric that
> tracks the number of generated PIDs per clientId. This will help the team
> who runs the Kafka cluster to
> - contact these rough clients and ask them to fix their clients or upgrade
> to a new client if the new client version has a better experience.
> - or if ended with a throttling solution this may help identify which
> clientId needs to be throttled.
>
> Maybe we can start with a solution for identifying the rough clients first
> and keep looking for a solution to limit them, what do you think?
>
> Thanks
>
> On Tue, Oct 18, 2022 at 5:24 PM Justine Olshan
>  wrote:
>
>> Oops.  I realized I just replied to Omnia 臘‍♀️
>>
>> Here was my response for the mailing thread:
>>
>> Hey Omnia,
>> Sorry to hear this is a problem for you as well. :(
>> > * I have some concerns about the impact of this option on the
>> transactional producers, for example, 

[jira] [Created] (KAFKA-14674) Backport fix for KAFKA-14455 to 3.3 and 3.4 branches

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14674:
-

 Summary: Backport fix for KAFKA-14455 to 3.3 and 3.4 branches
 Key: KAFKA-14674
 URL: https://issues.apache.org/jira/browse/KAFKA-14674
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton
 Fix For: 3.4.1, 3.3.3


The 3.4 branch is currently frozen for the upcoming 3.4.0 release. Once the 
release is complete, we can and should backport the change in 
[https://github.com/apache/kafka/pull/12984] onto the 3.3 and 3.4 branches 
before putting out any subsequent releases on those branches.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14673) High watermark listener

[David Jacot (Jira)]

David Jacot created KAFKA-14673:
---

 Summary: High watermark listener
 Key: KAFKA-14673
 URL: https://issues.apache.org/jira/browse/KAFKA-14673
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-888: Batch describe ACLs and describe client quotas

[Federico Valeri]

Hi Michael, thanks for the KIP.

In addition to the deduplication logic, I wonder if we should also
consider introducing a pagination mechanism. Talking about ACLs, there
could be many rules for each user and we may have lots of users on big
clusters.



On Tue, Jan 31, 2023 at 3:58 PM Mickael Maison  wrote:
>
> Hi Tom,
>
> Thanks for the feedback, you raised a few interesting points.
>
> Regarding the response size issue: With some APIs, if the same filter
> is in the request multiple times, the response will only include it
> once. To do so you need to include the filter in the response. For
> example, this works with describeConsumerGroups because the filter,
> the group id, is small. For ACLs and Quotas, I thought having the
> exact same filters multiple times in the requests should be rare so I
> privileged returning one entry for each filter in the request, and
> avoid including the filters back in the response.
>
> One option to limit potential abuse is to restrict the accepted
> filters. For example brokers could reject requests containing
> duplicate filters. Going further we could restrict the types of
> filters allowed and only allow requests with multiple filters to have
> multiple exact match filters, or if using fuzzy matches only allow it
> on different types. I don't think this would limit the usability of
> these APIs too much.
>
> Regarding using ids for each ACL: Yes StandardAuthorizer identifies
> unique ACLs with Uuids but custom Authorizer implementations don't
> necessarily do that. So while this could help reduce response size
> this would require significant changes to the Authorizer interface.
>
> Thanks,
> Mickael
>
>
>
>
> On Fri, Dec 2, 2022 at 6:55 PM Tom Bentley  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP. I can understand the motivation, but to be honest I
> > have a doubt about this.
> >
> > Taking the ACLs first, by allowing multiple filters with the current
> > proposal isn't there the chance that the same ACL will match multiple
> > filters, and be returned multiple times in the response. In fact, in the
> > worst case couldn't the client (by intent or accident) just request all
> > ACLs be included in the response an arbitrary number of times? This could
> > result in some pretty large responses. One way to avoid inflating the
> > response like this would be for the broker to deduplicate the ACLs in the
> > response by assigning an id for each, serialising each once and then using
> > the id to enumerate the matches for each pattern. It's worth noting that
> > the AccessControlEntryRecord used for KRaft clusters already has a Uuid. So
> > for the KRaft case it might be possible to use this, rather than the broker
> > having to deduplicate when handing the request.
> >
> > Another wrinkle is that if the broker calls
> > Authorizer.acls(AclBindingFilter) once for each pattern there's no
> > guarantee that the results are consistent (they could be modified between
> > calls). It could be made consistent with appropriate locking, but since in
> > practice this would be a very rare occurrence maybe we could just document
> > that as a limitation.
> >
> > Thanks again,
> >
> > Tom
> >
> > On Fri, 18 Nov 2022 at 18:00, Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > I have opened KIP-888 to allow describing ACLs and Quotas in batches:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-888%3A+Batch+describe+ACLs+and+describe+client+quotas
> > >
> > > Let me know if you have any feedback or suggestions.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1552

[Apache Jenkins Server]

See 

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Mickael Maison]

Hi Cheng Pan,

Are you still interested in following up with this KIP? This is a
useful feature so it would be nice to get it in Kafka.

Thanks,
Mickael

On Wed, Aug 17, 2022 at 1:39 PM Mickael Maison  wrote:
>
> Hi Cheng Pan,
>
> Thanks for the updates. I have a few more questions:
>
> 1) Since this KIP is targeting JsonConverter, I think we should add
> this new config to JsonConverterConfig instead of ConverterConfig.
>
> 2) What about renaming the new field to something like
> "use.null.for.optional.fields"?
>
> Thanks,
> Mickael
>
> On Thu, May 5, 2022 at 3:19 PM Cheng Pan  wrote:
> >
> > Update PR links
> >
> > [1] https://github.com/apache/kafka/pull/12126
> >
> > On 2022/05/05 13:16:59 Cheng Pan wrote:
> > > Hi Mickael,
> > >
> > > Thanks for ping me.
> > >
> > > I updated the KIP-581 description to narrow the change scope to address 
> > > this specific issue, and raised a WIP PR[1] to implement it.
> > >
> > > Please let me know if you have any concerns.
> > >
> > > [1] https://github.com/apache/kafka/pull/12126
> > >
> > > Thanks,
> > > Cheng Pan
> > >
> > > On 2022/05/02 12:57:25 Mickael Maison wrote:
> > > > Hi Cheng Pan,
> > > >
> > > > Thanks for raising this KIP, this would be a useful improvement!
> > > >
> > > > You never started a VOTE thread for this KIP. Are you interested in
> > > > finishing up this work?
> > > > If so, based on the discussion, I think you can open a VOTE thread as
> > > > described in 
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
> > > > If not, it's not a problem, someone else can volunteer to pick it up.
> > > >
> > > > Please let us know.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Sat, Aug 8, 2020 at 11:05 AM Ruslan Gibaiev  
> > > > wrote:
> > > > >
> > > > > Hello guys.
> > > > > Proposed PR seems to be fixing the issue in a backward-compatible way.
> > > > > Let's please move forward with it. Would be great to see it included 
> > > > > into next Kafka release
> > > > > Thank you
> > > > >
> > > > > On 2020/07/29 02:49:07, "379377...@qq.com" <379377...@qq.com> wrote:
> > > > > > Hi Chris,
> > > > > >
> > > > > > Thanks for your good suggestion, the KIP document and draft PR has 
> > > > > > been updated, please review again.
> > > > > >
> > > > > > And I found due to my misoperation, the mail thread has been 
> > > > > > broken, no idea how to fix it.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Cheng Pan
> > > > > >
> > > > > > From: Christopher Egerton
> > > > > > Date: 2020-05-04 10:53
> > > > > > To: dev
> > > > > > Subject: Re: [Discuss] KIP-581: Value of optional null field which 
> > > > > > has default value
> > > > > > Hi Cheng,
> > > > > >
> > > > > > I think refactoring that method should be fine (if maybe a little 
> > > > > > painful);
> > > > > > the method itself is private and all places that it's invoked 
> > > > > > directly are
> > > > > > either package-private or non-static, so it shouldn't affect any of 
> > > > > > the
> > > > > > public methods of the JSON converter to change "convertToConnect" 
> > > > > > to be
> > > > > > non-static. Even if it did, the only parts of the JSON converter 
> > > > > > that are
> > > > > > public API (and therefore officially subject to concerns about
> > > > > > compatibility) are the methods it implements that satisfy the 
> > > > > > "Converter"
> > > > > > and "HeaderConverter" interfaces.
> > > > > >
> > > > > > Would you mind explicitly specifying in the KIP that the new 
> > > > > > property will
> > > > > > be added for the JSON converter only, and that it will affect both
> > > > > > serialization and deserialization?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Apr 28, 2020 at 10:52 AM 379377944 <379377...@qq.com> wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > >
> > > > > > > Thanks for your reminder, the original implement is deprecated, I 
> > > > > > > just
> > > > > > > update the JIRA with the new
> > > > > > > PR link:  https://github.com/apache/kafka/pull/8575
> > > > > > >
> > > > > > >
> > > > > > > As question 2), I agree with you that we should consider both
> > > > > > > serialization and deserialization, and as you said, I only 
> > > > > > > implement the
> > > > > > > serialization now. This is  because the original serde implement 
> > > > > > > is not
> > > > > > > symmetrical, the convertToConnect is a static method and can’t 
> > > > > > > access the
> > > > > > > field in JsonConverter
> > > > > > > instance, maybe I should do some refactoring to implement the
> > > > > > > deserialization.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Cheng Pan
> > > > > > >  Original Message
> > > > > > > Sender: Christopher Egerton
> > > > > > > Recipient: dev
> > > > > > > Date: Wednesday, Apr 15, 2020 02:28
> > > > > > > Subject: Re: [Discuss] KIP-581: Value of optional null 

Re: Requesting permissions to contribute

[Mickael Maison]

Hi Nelson.

I granted you permissions on Jira and on the wiki.

Thanks,
Mickael

On Thu, Feb 2, 2023 at 10:33 AM Nelson Bighetti  wrote:
>
> Hello,
>
> My name is Nelson. I want to start contributing to the Apache Kafka project.
>
> JiraID: bachmanity1
> WikiID: bachmanity1
> email: bachmanity...@gmail.com
>
> Best regards,

Requesting permissions to contribute

[Nelson Bighetti]

Hello,

My name is Nelson. I want to start contributing to the Apache Kafka project.

JiraID: bachmanity1
WikiID: bachmanity1
email: bachmanity...@gmail.com

Best regards,