[jira] [Created] (KAFKA-13673) disable idempotent producer when acks/retries/max.in.flight config conflicts

Luke Chen created KAFKA-13673: - Summary: disable idempotent producer when acks/retries/max.in.flight config conflicts Key: KAFKA-13673 URL: https://issues.apache.org/jira/browse/KAFKA-13673 Project:

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

> And if I understood correctly, this is already done for COOPERATIVE anyway. Right. We'll keep the partitions status for prev-owned partitions in Cooperative protocol. Thanks. Luke On Thu, Feb 17, 2022 at 2:20 PM Ryan Leslie (BLOOMBERG/ 919 3RD A) < rles...@bloomberg.net> wrote: > Thanks

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

Thanks Luke. I’ve always passed a duration of 0 if polling while paused. The logic being that the poll is only used to preserve group membership and there’s no need to wait for anything. That probably explains why I would have been unlikely to ever see this behavior. I agree with changing the

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #701

See

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

Hi Artem, Also, one more thing I think you need to know. As this bug KAFKA-7572 mentioned, sometimes the custom partitioner would return negative partition id accidentally. If it returned -1, how could you know if it is expected or not expected?

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

Hi Ryan, > I've used pause() between version 0.10 and 2.7 with EAGER protocol only and have not yet observed cases where poll() returned messages unless resume() had been called. Yes, this issue only happens when group rebalance is triggered, the pause flag will be cleared. And if the

[jira] [Resolved] (KAFKA-13669) Log messages for source tasks with no offsets to commit are noisy and confusing

[ https://issues.apache.org/jira/browse/KAFKA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-13669. - Fix Version/s: 3.2.0 Resolution: Fixed > Log messages for source tasks with no

Re: [DISCUSS] Apache Kafka 3.2.0 release

Hi Bruno, Thank you for driving the release ! Can we add KIP-764 to the plan? ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor ) Thanks, 2022年2月16日(水) 18:22 Bruno Cadonna : > Hi Julien, > > Thank you for the feedback on the release

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #700

See Changes: -- [...truncated 442487 lines...] [2022-02-16T23:24:34.099Z] [2022-02-16T23:24:34.099Z]

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #699

See

[jira] [Created] (KAFKA-13672) Flaky test kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()

Bruno Cadonna created KAFKA-13672: - Summary: Flaky test kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize() Key: KAFKA-13672 URL: https://issues.apache.org/jira/browse/KAFKA-13672

Re: [DISCUSSION] New broker metric. Per partition consumer offset

Hello, Dylan. > At larger scales (e.g., thousands+ of partitions and hundreds+ of consumer > groups) the cardinality of metrics is very high for a broker and very > challenging for a metrics collector to pull out of JMX. Agreed. 0. Kafka have `metrics.jmx.exclude`, `metrics.jmx.include`

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #698

See

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

Thank you Matthias, this is great feedback. Adding my comments below. On Wed, 16 Feb 2022 at 00:42, Matthias J. Sax wrote: > Thanks for the KIP. > > In alignment to my reply to KIP-634, I am wondering if we are heading > into the right direction, or if we should consider to re-design the DSL >

Re: [DISCUSSION] New broker metric. Per partition consumer offset

It would be very convenient for consumer applications that are not collecting and shipping their own metrics to have Kafka cluster doing this for them. At larger scales (e.g., thousands+ of partitions and hundreds+ of consumer groups) the cardinality of metrics is very high for a broker and

Re: [DISCUSSION] New broker metric. Per partition consumer offset

Chris, thanks for the support. Dear Kafka committers, can you, please, advise me: Are you support my proposal? Can I implement new metrics in the scope of separate KIP? KIP-714 seems to me much more complex improvement. Moreover, it has similar but slightly different goal. All I propose is to

Re: [DISCUSSION] New broker metric. Per partition consumer offset

Hi Nikolay, Yep, makes sense to me  Sounds like the motivation here is similar to KIP-714 [1], which allows clients to publish their own metrics directly to a broker. It seems like one reason this use case isn't already addressed in that KIP is that, if all you're doing is taking the delta

[jira] [Created] (KAFKA-13671) Power (ppc64le) support for kafka

Abhijit created KAFKA-13671: --- Summary: Power (ppc64le) support for kafka Key: KAFKA-13671 URL: https://issues.apache.org/jira/browse/KAFKA-13671 Project: Kafka Issue Type: Improvement

Re: [DISCUSSION] New broker metric. Per partition consumer offset

Hello, Chris. Thanks for the feedback. > Have you seen the consumer-side lag metrics [1]? "records-lag», Yes, I’m aware of these metrics. > If so, I'd be curious to know what the motivation for duplicating existing > client metrics onto brokers would be? It can be a complex task to setup and

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #697

See Changes: -- [...truncated 444358 lines...] [2022-02-16T14:12:00.384Z] [INFO] [2022-02-16T14:12:00.384Z] [INFO] --- maven-archetype-plugin:2.2:update-local-catalog

Re: [DISCUSSION] New broker metric. Per partition consumer offset

Hi Nikolay, Have you seen the consumer-side lag metrics [1]? "records-lag", "records-lag-avg", "records-lag-max" all give lag stats on a per-topic-partition basis. If so, I'd be curious to know what the motivation for duplicating existing client metrics onto brokers would be? [1]

[DISCUSSION] New broker metric. Per partition consumer offset

Hello, Kafka team. When running in production the common user question is «How big lag between producer and consumer?». We have a `kafka-consumer-groups.sh` tool and `AdminClient#getListConsumerGroupOffsetsCall` to answer the question. Even detailed guides on how to calculate *consumer lag*

Re: [DISCUSS] Apache Kafka 3.2.0 release

Hi Julien, Thank you for the feedback on the release plan. I added KIP-808 to the plan. Best, Bruno On 16.02.22 09:10, Julien Chanaud wrote: Hello Bruno, Can we also add KIP-808 to the plan ?

Re: [DISCUSS] Apache Kafka 3.2.0 release

Hi Mickael, Thank you for your feedback on the release plan! I added KIP-769 to the plan. Best, Bruno On 15.02.22 19:00, Mickael Maison wrote: Hi Bruno, Thanks for publishing the release plan! Can we add KIP-769 to the plan? Thanks, Mickael On Tue, Feb 15, 2022 at 12:37 PM Bruno Cadonna

Re: [DISCUSS] Apache Kafka 3.2.0 release

Hello Bruno, Can we also add KIP-808 to the plan ? https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+different+unix+precisions+in+TimestampConverter+SMT Thank you, Julien Le mar. 15 févr. 2022 à 19:01, Mickael Maison a écrit : > Hi Bruno, > > Thanks for publishing