[jira] [Created] (KAFKA-17550) Add INCONSISTENT_GROUP_TYPE to ConsumerGroupDescribe and kafka-consumer-groups.sh

2024-09-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-17550: Summary: Add INCONSISTENT_GROUP_TYPE to ConsumerGroupDescribe and kafka-consumer-groups.sh Key: KAFKA-17550 URL: https://issues.apache.org/jira/browse/KAFKA-17550

[jira] [Resolved] (KAFKA-17230) Kafka consumer client doesn't report node request-latency metrics

2024-08-01 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17230. - Fix Version/s: 3.9.0 Resolution: Fixed merged the PR to trunk > Kafka consumer client does

[jira] [Created] (KAFKA-17230) Kafka consumer client doesn't report node request-latency metrics

2024-07-31 Thread Apoorv Mittal (Jira)
Apoorv Mittal created KAFKA-17230: - Summary: Kafka consumer client doesn't report node request-latency metrics Key: KAFKA-17230 URL: https://issues.apache.org/jira/browse/KAFKA-17230 Project:

[jira] [Created] (KAFKA-17218) kafka-consumer-groups fails to describe all group if one group has been consuming from a deleted topic

2024-07-30 Thread Omnia Ibrahim (Jira)
Omnia Ibrahim created KAFKA-17218: - Summary: kafka-consumer-groups fails to describe all group if one group has been consuming from a deleted topic Key: KAFKA-17218 URL: https://issues.apache.org/jira/browse

Re: [DISCUSS] KIP-1048 Improve kafka-consumer-perf-test to benchmark single partition

2024-05-27 Thread Harsh Panchal
Bumping up this thread since I can not find it in the mail archive. On Wed, 22 May 2024 at 18:09, Harsh Panchal wrote: > Hi, > > I would like to propose a change in the kafka-consumer-perf-test tool to > support perf testing specific partitions. > > kafka-consumer-perf-test

[DISCUSS] KIP-1048 Improve kafka-consumer-perf-test to benchmark single partition

2024-05-22 Thread Harsh Panchal
Hi, I would like to propose a change in the kafka-consumer-perf-test tool to support perf testing specific partitions. kafka-consumer-perf-test is a great tool to quickly check raw consumer performance. Currently, It subscribes to all the partitions and gives overall cluster performance, however

[jira] [Created] (KAFKA-16810) Improve kafka-consumer-perf-test to benchmark single partition

2024-05-21 Thread Harsh Panchal (Jira)
Harsh Panchal created KAFKA-16810: - Summary: Improve kafka-consumer-perf-test to benchmark single partition Key: KAFKA-16810 URL: https://issues.apache.org/jira/browse/KAFKA-16810 Project: Kafka

[jira] [Created] (KAFKA-16309) Enabling virtual threads in the Kafka Consumer

2024-02-27 Thread Soby Chacko (Jira)
Soby Chacko created KAFKA-16309: --- Summary: Enabling virtual threads in the Kafka Consumer Key: KAFKA-16309 URL: https://issues.apache.org/jira/browse/KAFKA-16309 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16285) Make group metadata available when a new assignment is set in async Kafka consumer

2024-02-20 Thread Bruno Cadonna (Jira)
Bruno Cadonna created KAFKA-16285: - Summary: Make group metadata available when a new assignment is set in async Kafka consumer Key: KAFKA-16285 URL: https://issues.apache.org/jira/browse/KAFKA-16285

[jira] [Created] (KAFKA-16248) Kafka consumer should cache leader offset ranges

2024-02-13 Thread Lucas Brutschy (Jira)
Lucas Brutschy created KAFKA-16248: -- Summary: Kafka consumer should cache leader offset ranges Key: KAFKA-16248 URL: https://issues.apache.org/jira/browse/KAFKA-16248 Project: Kafka Issue

RE: Kafka consumer group crashing and not able to consume once service is up

2024-02-09 Thread Marigowda, Santhosh Aditya
; Namboodiri, Vishnu ; Mudlapur, Rajesh ; Reddy, Thanmai Subject: RE: Kafka consumer group crashing and not able to consume once service is up Hi Philip, Are you expecting logs from subscriber from our service please do let me know. Thanks, Santhosh Aditya From: Marigowda, Santhosh Aditya Sent

RE: Kafka consumer group crashing and not able to consume once service is up

2024-02-08 Thread Marigowda, Santhosh Aditya
, Thanmai Subject: RE: Kafka consumer group crashing and not able to consume once service is up Hi Philip, Please find the service logs(Subscriber logs). We don’t see any logs-related issues with consumer. Kafka consumer configuration kafka-consumer = { server= "10.221.10

Re: Kafka consumer group crashing and not able to consume once service is up

2024-02-08 Thread Philip Nee
..@in.unisys.com>; Mudlapur, Rajesh < > rajesh.mudla...@au.unisys.com>; Reddy, Thanmai ; > Marigowda, Santhosh Aditya > *Subject:* RE: Kafka consumer group crashing and not able to consume once > service is up > > > > Hi Philip, > > Thanks for your queries, Please

Re: Kafka consumer group crashing and not able to consume once service is up

2024-02-07 Thread Philip Nee
Hi Santhosh, Your problem statement confuses me a bit (apologize). You mentioned "if one of the kafka consumer(Service)" - Do you have a single member consumer group? Could you elaborate on the setup a bit? Did you also mean after restarting the "service", the service was not

RE: Kafka consumer group crashing and not able to consume once service is up

2024-02-07 Thread Marigowda, Santhosh Aditya
: Matthias J. Sax mailto:mj...@apache.org>> Sent: Wednesday, January 31, 2024 12:13 AM To: dev@kafka.apache.org<mailto:dev@kafka.apache.org> Subject: Re: Kafka consumer group crashing and not able to consume once service is up I am a not sure if I can follow completely. From the figures you sh

RE: Kafka consumer group crashing and not able to consume once service is up

2024-01-30 Thread Marigowda, Santhosh Aditya
@kafka.apache.org Subject: Re: Kafka consumer group crashing and not able to consume once service is up I am a not sure if I can follow completely. From the figures you show, you have a topic with 4 partitions, and 4 consumer groups. Thus, each consumer group should read all 4 partitions, but the

Re: Kafka consumer group crashing and not able to consume once service is up

2024-01-30 Thread Matthias J. Sax
with our problem. In our POC, if one of the kafka consumer(Service) shuts down or crashes then post restart of service none of the messages are getting consumed by the crashed Service. Other services are consuming without any issues. One of service crash/Shutdown If we rename the Kafka con

Review request for Java Kafka consumer

2023-06-26 Thread Erik van Oosten
Dear Kafka developers, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka consumer on the JVM is not usable from asynchronous runtimes such as Kotlin co-routines and ZIO. Your review is much appreciated. Kind regards,     Erik. -- Erik van

[jira] [Created] (KAFKA-14878) Reduce assignment data size to improve Kafka Consumer scalability

2023-04-04 Thread Armand (Jira)
Armand created KAFKA-14878: -- Summary: Reduce assignment data size to improve Kafka Consumer scalability Key: KAFKA-14878 URL: https://issues.apache.org/jira/browse/KAFKA-14878 Project: Kafka Issue

[jira] [Created] (KAFKA-14739) Kafka consumer reading messages out of order after a rebalance

2023-02-21 Thread Colin Shaw (Jira)
Colin Shaw created KAFKA-14739: -- Summary: Kafka consumer reading messages out of order after a rebalance Key: KAFKA-14739 URL: https://issues.apache.org/jira/browse/KAFKA-14739 Project: Kafka

[jira] [Created] (KAFKA-14626) Kafka Consumer Coordinator does not cleanup all metrics

2023-01-17 Thread Feiyan Yu (Jira)
Feiyan Yu created KAFKA-14626: - Summary: Kafka Consumer Coordinator does not cleanup all metrics Key: KAFKA-14626 URL: https://issues.apache.org/jira/browse/KAFKA-14626 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14366) Kafka consumer rebalance issue, offsets points back to very old committed offset

2022-11-07 Thread Chetan (Jira)
Chetan created KAFKA-14366: -- Summary: Kafka consumer rebalance issue, offsets points back to very old committed offset Key: KAFKA-14366 URL: https://issues.apache.org/jira/browse/KAFKA-14366 Project: Kafka

[jira] [Resolved] (KAFKA-14161) kafka-consumer-group.sh --list not list all consumer groups

2022-08-11 Thread Luke Chen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-14161. --- Resolution: Not A Problem > kafka-consumer-group.sh --list not list all consumer gro

[jira] [Created] (KAFKA-14161) kafka-consumer-group.sh --list not list all consumer groups

2022-08-11 Thread yandufeng (Jira)
yandufeng created KAFKA-14161: - Summary: kafka-consumer-group.sh --list not list all consumer groups Key: KAFKA-14161 URL: https://issues.apache.org/jira/browse/KAFKA-14161 Project: Kafka Issue

[jira] [Created] (KAFKA-13961) GKE kafka-consumer-groups returns incorrect hosts

2022-06-06 Thread Mikhail Filatov (Jira)
Mikhail Filatov created KAFKA-13961: --- Summary: GKE kafka-consumer-groups returns incorrect hosts Key: KAFKA-13961 URL: https://issues.apache.org/jira/browse/KAFKA-13961 Project: Kafka

Kafka Consumer Partition Non Uniform Data Fetch/Pull

2022-05-11 Thread Muthuswamy S
some partitions over others. Our messages are different in sizes and we have set max bytes per message and overall partition fetch size too. How can we ensure to process Round Robin mode in Kafka Consumer with the partitions assigned to it? Can you pls help? -- thanks, Muthuswamy.S

Re: New Kafka Consumer : unknown member id

2021-11-05 Thread Kafka Life
Hello Luke i have build a new kafka environment with kafka 2.8.0 the consumer is a new consumer set up to this environment is throwing the below error... the old consumers for the same applications for the same environment -2.8.0 is working fine.. . could you please advise 2021-11-02 12:25:24 D

Re: New Kafka Consumer : unknown member id

2021-10-28 Thread Luke Chen
Hi, Which version of kafka client are you using? I can't find this error message in the source code. When googling this error message, it showed the error is in Kafka v0.9. Could you try to use the V3.0.0 and see if that issue still exist? Thank you. Luke On Thu, Oct 28, 2021 at 11:15 PM Kafka L

New Kafka Consumer : unknown member id

2021-10-28 Thread Kafka Life
Dear Kafka Experts We have set up a group.id (consumer ) = YYY But when tried to connect to kafka instance : i get this error message. I am sure this consumer (group id does not exist in kafka) .We user plain text protocol to connect to kafka 2.8.0. Please suggest how to resolve this issue. D

[jira] [Created] (KAFKA-12926) ConsumerGroupCommand's java.lang.NullPointerException at negative offsets while running kafka-consumer-groups.sh

2021-06-09 Thread Ignacio Acuna (Jira)
Ignacio Acuna created KAFKA-12926: - Summary: ConsumerGroupCommand's java.lang.NullPointerException at negative offsets while running kafka-consumer-groups.sh Key: KAFKA-12926 URL: https://issues.apache.org

[DISCUSS] KIP-748 Kafka Consumer Record Latency Metric

2021-06-02 Thread Ryan Dielhenn
Hey kafka-dev, I created KIP-748 as a proposal to add broker count metrics to the Quorum Controller. https://cwiki.apache.org/confluence/display/KAFKA/KIP-748%3A+Add+Broker+Count+Metrics#KIP748:AddBrokerCountMetrics Best, Ryan Dielhenn

[jira] [Created] (KAFKA-12821) Kafka consumer skip reading from single partition

2021-05-20 Thread Prakash Patel (Jira)
Prakash Patel created KAFKA-12821: - Summary: Kafka consumer skip reading from single partition Key: KAFKA-12821 URL: https://issues.apache.org/jira/browse/KAFKA-12821 Project: Kafka Issue

Re: Kafka Consumer Incremental Rebalance Protocol | Downgrade

2021-05-11 Thread Vipul Goyal
cond rolling > bounce in which > you remove the CooperativeStickyAssignor and downgrade the member to 2.3 > again. It's just > the inverted upgrade path. > > Hope that helps, > Sophie > > On Tue, May 11, 2021 at 7:36 AM Vipul Goyal > wrote: > > > Hi Tea

Re: Kafka Consumer Incremental Rebalance Protocol | Downgrade

2021-05-11 Thread Sophie Blee-Goldman
ipul Goyal wrote: > Hi Team, > > I need some guidance related to Kafka Consumer Incremental Rebalance > Protocol. > > I was following the below KIP and understood the upgrade path, But a bit > confused with the downgrade procedure. > > *KIP:* > > https://cwiki.apac

Kafka Consumer Incremental Rebalance Protocol | Downgrade

2021-05-11 Thread Vipul Goyal
Hi Team, I need some guidance related to Kafka Consumer Incremental Rebalance Protocol. I was following the below KIP and understood the upgrade path, But a bit confused with the downgrade procedure. *KIP:* https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental

[jira] [Created] (KAFKA-12605) kafka consumer churns through buffer memory iterating over records

2021-04-01 Thread radai rosenblatt (Jira)
radai rosenblatt created KAFKA-12605: Summary: kafka consumer churns through buffer memory iterating over records Key: KAFKA-12605 URL: https://issues.apache.org/jira/browse/KAFKA-12605 Project

[jira] [Resolved] (KAFKA-12428) Add a last-heartbeat-seconds-ago metric to Kafka Consumer

2021-03-29 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-12428. --- Resolution: Duplicate > Add a last-heartbeat-seconds-ago metric to Kafka Consu

[jira] [Created] (KAFKA-12531) kafka-consumer-groups list got

2021-03-23 Thread donglei (Jira)
donglei created KAFKA-12531: --- Summary: kafka-consumer-groups list got Key: KAFKA-12531 URL: https://issues.apache.org/jira/browse/KAFKA-12531 Project: Kafka Issue Type: Bug Affects Versions

[jira] [Created] (KAFKA-12428) Add a last-heartbeat-seconds-ago metric to Kafka Consumer

2021-03-04 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-12428: - Summary: Add a last-heartbeat-seconds-ago metric to Kafka Consumer Key: KAFKA-12428 URL: https://issues.apache.org/jira/browse/KAFKA-12428 Project: Kafka

[jira] [Created] (KAFKA-10708) Add "group-id" Tag to Kafka Consumer Metrics

2020-11-11 Thread Brian Wyka (Jira)
Brian Wyka created KAFKA-10708: -- Summary: Add "group-id" Tag to Kafka Consumer Metrics Key: KAFKA-10708 URL: https://issues.apache.org/jira/browse/KAFKA-10708 Project: Kafka

[jira] [Created] (KAFKA-10685) --to-datetime passed to kafka-consumer-groups getting interpreted as a timezone

2020-11-04 Thread Russell Sayers (Jira)
Russell Sayers created KAFKA-10685: -- Summary: --to-datetime passed to kafka-consumer-groups getting interpreted as a timezone Key: KAFKA-10685 URL: https://issues.apache.org/jira/browse/KAFKA-10685

Kafka consumer Exception handling and offset management

2020-09-23 Thread Preethkumar Thirupathi
Hi Team, Currently I'm implementing Spring boot @kafkalistener. I need to know that what all are the exception that may raise, while try to consume the record from kafka topic. How to commit the current offset after successfully processed the record.In between that I have handle the exception. Kin

[jira] [Created] (KAFKA-10422) Provide a `timesForOffsets` operation in kafka consumer

2020-08-20 Thread Guillaume Bort (Jira)
Guillaume Bort created KAFKA-10422: -- Summary: Provide a `timesForOffsets` operation in kafka consumer Key: KAFKA-10422 URL: https://issues.apache.org/jira/browse/KAFKA-10422 Project: Kafka

[jira] [Reopened] (KAFKA-10134) High CPU issue during rebalance in Kafka consumer after upgrading to 2.5

2020-08-13 Thread Ismael Juma (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-10134: - > High CPU issue during rebalance in Kafka consumer after upgrading to

[jira] [Resolved] (KAFKA-10134) High CPU issue during rebalance in Kafka consumer after upgrading to 2.5

2020-07-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-10134. --- Resolution: Fixed > High CPU issue during rebalance in Kafka consumer after upgrading

[jira] [Created] (KAFKA-10134) High CPU issue during rebalance in Kafka consumer after upgrading to 2.5

2020-06-09 Thread Sean Guo (Jira)
Sean Guo created KAFKA-10134: Summary: High CPU issue during rebalance in Kafka consumer after upgrading to 2.5 Key: KAFKA-10134 URL: https://issues.apache.org/jira/browse/KAFKA-10134 Project: Kafka

[jira] [Created] (KAFKA-10007) Kafka consumer offset reset despite recent group activity

2020-05-15 Thread Raman Gupta (Jira)
Raman Gupta created KAFKA-10007: --- Summary: Kafka consumer offset reset despite recent group activity Key: KAFKA-10007 URL: https://issues.apache.org/jira/browse/KAFKA-10007 Project: Kafka

[jira] [Resolved] (KAFKA-6822) Kafka Consumer 0.10.2.1 can not normally read data from Kafka 1.0.0

2020-05-14 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6822. Resolution: Abandoned > Kafka Consumer 0.10.2.1 can not normally read data from Ka

[jira] [Created] (KAFKA-9902) java client api can not completely take out the kafka-consumer-groups.sh output of information

2020-04-22 Thread startjava (Jira)
startjava created KAFKA-9902: Summary: java client api can not completely take out the kafka-consumer-groups.sh output of information Key: KAFKA-9902 URL: https://issues.apache.org/jira/browse/KAFKA-9902

[jira] [Created] (KAFKA-9775) IllegalFormatConversionException from kafka-consumer-perf-test.sh

2020-03-27 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9775: -- Summary: IllegalFormatConversionException from kafka-consumer-perf-test.sh Key: KAFKA-9775 URL: https://issues.apache.org/jira/browse/KAFKA-9775 Project: Kafka

[jira] [Created] (KAFKA-9761) kafka-consumer-groups tool overrides admin client defaults with a short 5 s timeout

2020-03-25 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9761: --- Summary: kafka-consumer-groups tool overrides admin client defaults with a short 5 s timeout Key: KAFKA-9761 URL: https://issues.apache.org/jira/browse/KAFKA-9761

[jira] [Created] (KAFKA-9646) kafka consumer cause high cpu usage

2020-03-04 Thread li xiangyuan (Jira)
li xiangyuan created KAFKA-9646: --- Summary: kafka consumer cause high cpu usage Key: KAFKA-9646 URL: https://issues.apache.org/jira/browse/KAFKA-9646 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-9306) Kafka Consumer does not clean up all metrics after shutdown

2020-02-18 Thread Sanjana Kaundinya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjana Kaundinya resolved KAFKA-9306. -- Fix Version/s: 2.4.1 Resolution: Fixed > Kafka Consumer does not clean up

[jira] [Resolved] (KAFKA-5868) Kafka Consumer Rebalancing takes too long

2020-02-04 Thread Sophie Blee-Goldman (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-5868. Fix Version/s: 2.5.0 Resolution: Fixed > Kafka Consumer Rebalancing ta

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-17 Thread Sean Glover
gt; Consumer lag is a useful metric to monitor how many records > are > > > > > >> queued to > > > > > >> > > be processed. We can look at individual lag per partition > or we > > > may > > > > > >> > >

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-17 Thread Sean Glover
percentiles, but one aggregate that does not make sense is to sum latency across many partitions, because partitions are typically always consumed in parallel fashion. > > 2. Some existing solutions already expose the consumer group lap in time. > See > > https://www.lightbend.com/

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-15 Thread Habib Nahas
entify > > > > >> > > hot partitions, caused by an insufficient producing partitioning > > > > >> strategy. > > > > >> > > We may want to monitor a sum of lag across all partitions so we > > have a > > > > >> > >

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-14 Thread Sean Glover
gt; have a > > > >> > > sense as to our total backlog of messages to consume. Lag in > offsets > > > >> is > > > >> > > useful when you have a good understanding of your messages and > > > >> processing > > > >> >

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-14 Thread Habib Nahas
> >> > > characteristics, but it doesn’t tell us how far behind *in time* we > > >> are. > > >> > > This is known as wait time in queueing theory, or more informally > > >> > > it’s > > >> > > referred to as latency. > >

[jira] [Resolved] (KAFKA-8179) Incremental Rebalance Protocol for Kafka Consumer

2020-01-09 Thread Guozhang Wang (Jira)
nce Protocol for Kafka Consumer > - > > Key: KAFKA-8179 > URL: https://issues.apache.org/jira/browse/KAFKA-8179 > Project: Kafka > Issue Type: Improvement >

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-20 Thread Gokul Ramanan Subramanian
/monitor-kafka-consumer-group-latency-with-kafka-lag-exporter for an example. *The KIP should reference existing solutions and suggest the benefits of using the native solution that you propose*. 3. If a message was produced a long time ago, and a new consumer group has been created, then the

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-18 Thread Habib Nahas
a > >> > > consumer. The latency of records in a partition correlates with lag, > >> but a > >> > > larger lag doesn’t necessarily mean a larger latency. For example, a > >> topic > >> > > consumed by two separate application consumer groups A

[jira] [Created] (KAFKA-9306) Kafka Consumer does not clean up all metrics after shutdown

2019-12-16 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-9306: --- Summary: Kafka Consumer does not clean up all metrics after shutdown Key: KAFKA-9306 URL: https://issues.apache.org/jira/browse/KAFKA-9306 Project: Kafka

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Sean Glover
essing time is slower, it takes >> > > longer to process each message per partition. Meanwhile, Application >> B is >> > > a consumer which performs a simple ETL operation to land streaming >> data in >> > > another system, such as HDFS. It may have similar

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Sean Glover
ny consumer group members to > handle the > > > load quickly enough, but since its processing time is slower, it takes > > > longer to process each message per partition. Meanwhile, Application B > is > > > a consumer which performs a simple ETL operation to land stre

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Habib Nahas
es > > longer to process each message per partition. Meanwhile, Application B is > > a consumer which performs a simple ETL operation to land streaming data in > > another system, such as HDFS. It may have similar lag to Application A, but > > because it has a faster p

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-11 Thread Sean Glover
ime is slower, it takes > longer to process each message per partition. Meanwhile, Application B is > a consumer which performs a simple ETL operation to land streaming data in > another system, such as HDFS. It may have similar lag to Application A, but > because it has a faster processin

Handling deserialization exceptions in Kafka consumer

2019-12-02 Thread Sönke Liebau
Hi all, I was recently discussing deserialization errors and how they can be handled in Kafka consumers. I believe the current code still throws an exception if deserialization fails which stops consumption unless you seek past that record. This creates an issue for a JDBC sink connector that us

[jira] [Created] (KAFKA-9233) Kafka consumer throws undocumented IllegalStateException

2019-11-26 Thread Andrew Olson (Jira)
Andrew Olson created KAFKA-9233: --- Summary: Kafka consumer throws undocumented IllegalStateException Key: KAFKA-9233 URL: https://issues.apache.org/jira/browse/KAFKA-9233 Project: Kafka Issue

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-09-30 Thread Guozhang Wang
Hello folks, One last update on the KIP: we've added a section with a list of newly added metrics corresponding to consumer rebalance events as part of this proposal as well, detailed list can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Increm

[jira] [Created] (KAFKA-8898) if there is no message for poll, kafka consumer apply memory

2019-09-11 Thread linking12 (Jira)
linking12 created KAFKA-8898: Summary: if there is no message for poll, kafka consumer apply memory Key: KAFKA-8898 URL: https://issues.apache.org/jira/browse/KAFKA-8898 Project: Kafka Issue

How to test the performance of Kafka consumer?

2019-09-06 Thread 郑立豹
I want to test the throughput and latency of Kafka consumer through the application, but there is no test entry. How to test and collect throughput and latency data?

[jira] [Created] (KAFKA-8883) Kafka Consumer continuously displaying info message Seeking to offset 0 for partition

2019-09-06 Thread Veerabhadra Rao Mallavarapu (Jira)
Veerabhadra Rao Mallavarapu created KAFKA-8883: -- Summary: Kafka Consumer continuously displaying info message Seeking to offset 0 for partition Key: KAFKA-8883 URL: https://issues.apache.org/jira

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-08-05 Thread Guozhang Wang
avaiable > > > > > in consumers. > > > > > > > > > > > > Thanks, > > > > > > Harsha > > > > > > > > > > > > On Wed, May 22, 2019, at 12:24 AM, Liquan Pei wrote: > > > > > >> +1 (

Re: Kafka consumer is not reading some partition

2019-07-24 Thread Omar Al-Safi
Hi, Maybe it would be helpful if you can attach the logs for your consumers when you notice they stopped consuming from some of your partitions On Tue, 23 Jul 2019 at 16:56, Sergey Fedorov wrote: > Hello. I was using Kafka 2.1.1 and facing a problem where our consumers > sometimes intermittentl

Kafka consumer is not reading some partition

2019-07-23 Thread Sergey Fedorov
Hello. I was using Kafka 2.1.1 and facing a problem where our consumers sometimes intermittently stop consuming from one or two of the partitions. My config

[jira] [Created] (KAFKA-8697) Kafka consumer group auto removal

2019-07-23 Thread Pavel Rogovoy (JIRA)
Pavel Rogovoy created KAFKA-8697: Summary: Kafka consumer group auto removal Key: KAFKA-8697 URL: https://issues.apache.org/jira/browse/KAFKA-8697 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-07-15 Thread Sean Glover
antly less. If the Kafka Consumer reported a latency metric it would be easier to build Service Level Agreements (SLAs) based on non-functional requirements of the streaming system. For example, the system must never have a latency of greater than 10 minutes. This SLA could be used in monitoring a

[DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-07-11 Thread Sean Glover
Hi kafka-dev, I've created KIP-489 as a proposal for adding latency metrics to the Kafka Consumer in a similar way as record-lag metrics are implemented. https://cwiki.apache.org/confluence/display/KAFKA/489%3A+Kafka+Consumer+Record+Latency+Metric Regards, Sean -- Principal Eng

[jira] [Created] (KAFKA-8656) Kafka Consumer Record Latency Metric

2019-07-11 Thread Sean Glover (JIRA)
Sean Glover created KAFKA-8656: -- Summary: Kafka Consumer Record Latency Metric Key: KAFKA-8656 URL: https://issues.apache.org/jira/browse/KAFKA-8656 Project: Kafka Issue Type: New Feature

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-06-28 Thread Sophie Blee-Goldman
> > > > >> > > > > >> On Tue, May 21, 2019 at 11:34 PM Boyang Chen > > > > > wrote: > > > > >> > > > > >>> Thank you Guozhang for all the hard work. > > > > >>> > > > > >&

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-06-27 Thread Boyang Chen
2019, at 12:24 AM, Liquan Pei wrote: > > > >> +1 (non-binding) > > > >> > > > >> On Tue, May 21, 2019 at 11:34 PM Boyang Chen > > > wrote: > > > >> > > > >>> Thank you Guozhang for all the hard work. > > > >>> > >

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-06-27 Thread Sophie Blee-Goldman
Liquan Pei wrote: > > >> +1 (non-binding) > > >> > > >> On Tue, May 21, 2019 at 11:34 PM Boyang Chen > > wrote: > > >> > > >>> Thank you Guozhang for all the hard work. > > >>> > > >>> +

[jira] [Created] (KAFKA-8506) Kafka Consumer broker never stops on connection failure

2019-06-07 Thread Gaurav (JIRA)
Gaurav created KAFKA-8506: - Summary: Kafka Consumer broker never stops on connection failure Key: KAFKA-8506 URL: https://issues.apache.org/jira/browse/KAFKA-8506 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-30 Thread Guozhang Wang
t; >> On Tue, May 21, 2019 at 11:34 PM Boyang Chen > wrote: > >> > >>> Thank you Guozhang for all the hard work. > >>> > >>> +1 (non-binding) > >>> > >>> > >>> From: Guozha

[jira] [Created] (KAFKA-8431) Add a onTimeoutExpired callback to Kafka Consumer

2019-05-26 Thread Richard Yu (JIRA)
Richard Yu created KAFKA-8431: - Summary: Add a onTimeoutExpired callback to Kafka Consumer Key: KAFKA-8431 URL: https://issues.apache.org/jira/browse/KAFKA-8431 Project: Kafka Issue Type

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-22 Thread Matthias J. Sax
21, 2019 at 11:34 PM Boyang Chen wrote: >> >>> Thank you Guozhang for all the hard work. >>> >>> +1 (non-binding) >>> >>> >>> From: Guozhang Wang >>> Sent: Wednesday, May 22, 2019 1:32 A

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-22 Thread Harsha
d work. > > > > +1 (non-binding) > > > > > > From: Guozhang Wang > > Sent: Wednesday, May 22, 2019 1:32 AM > > To: dev > > Subject: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol > > > > Hel

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-22 Thread Liquan Pei
+1 (non-binding) On Tue, May 21, 2019 at 11:34 PM Boyang Chen wrote: > Thank you Guozhang for all the hard work. > > +1 (non-binding) > > > From: Guozhang Wang > Sent: Wednesday, May 22, 2019 1:32 AM > To: dev > Subject: [VO

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-21 Thread Boyang Chen
Thank you Guozhang for all the hard work. +1 (non-binding) From: Guozhang Wang Sent: Wednesday, May 22, 2019 1:32 AM To: dev Subject: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol Hello folks, I'd like to start the voting for KIP-42

[VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-21 Thread Guozhang Wang
Hello folks, I'd like to start the voting for KIP-429 now, details can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-RebalanceCallbackErrorHandling And the on-goin

Kafka Consumer - Random access fetching of partition range

2019-05-02 Thread Nitzan Aviram
Hi Kafka developers, *The question:* How can I randomly fetch an old chunk of messages with a given range definition of [partition, start offset, end offset]. Hopefully ranges from multiple partitions at once (one range for each partition). This needs to be supported in a concurrent environment to

[Jira][Created](KAFKA-8100)kafka consumer not refresh metadata for dynamic topic deletion

2019-04-15 Thread Shengnan YU
Hi everyone: I found that KafkaConsumer will not refresh its topic metadata when some unused topics are deleted, which leads to continuous UNKNOWN_TOPIC_PARTITION warning. In source code KafkaProducer will remove unused topic after a expire time but KafkaConsumer not. I know it may be a design t

[jira] [Created] (KAFKA-8179) Incremental Rebalance Protocol for Kafka Consumer

2019-04-01 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-8179: Summary: Incremental Rebalance Protocol for Kafka Consumer Key: KAFKA-8179 URL: https://issues.apache.org/jira/browse/KAFKA-8179 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group

2019-03-29 Thread Guozhang Wang (JIRA)
://issues.apache.org/jira/browse/KAFKA-5154 now. > Two instances of kafka consumer reading the same partition within a consumer > group > -- > > Key: KAFKA-6681 > URL: https://issues.apac

[jira] [Created] (KAFKA-8100) If delete expired topic, kafka consumer will keep flushing unknown_topic warning in log

2019-03-13 Thread Shengnan YU (JIRA)
Shengnan YU created KAFKA-8100: -- Summary: If delete expired topic, kafka consumer will keep flushing unknown_topic warning in log Key: KAFKA-8100 URL: https://issues.apache.org/jira/browse/KAFKA-8100

[jira] [Created] (KAFKA-7995) Augment singleton protocol type to list for Kafka Consumer

2019-02-24 Thread Boyang Chen (JIRA)
Boyang Chen created KAFKA-7995: -- Summary: Augment singleton protocol type to list for Kafka Consumer Key: KAFKA-7995 URL: https://issues.apache.org/jira/browse/KAFKA-7995 Project: Kafka

Kafka consumer issue

2019-01-07 Thread Vikas Talegaonkar
@Team, I am trying to connect and consume from remote kafka, i am able to get number of partitions but when tried to consume from it not getting any data either start consuming from beginning or from latest. We have multiple topic in this kafka cluster, and there are consumer running for other

Need help on how to commit kafka consumer with other threads

2018-12-13 Thread SriPrakash Sai
I implemented one application using @kafkaListener. All listened events will be kept in Redis for some time. WIth the help of Spring scheduler, above events will be spanned across multiple threads and processed some business logic. Can you suggest me how to commit the events to kafka after process

[jira] [Created] (KAFKA-7684) kafka consumer SchemaException occurred: Error reading field 'brokers':

2018-11-27 Thread leibo (JIRA)
leibo created KAFKA-7684: Summary: kafka consumer SchemaException occurred: Error reading field 'brokers': Key: KAFKA-7684 URL: https://issues.apache.org/jira/browse/KAFKA-7684 Proj

  1   2   3   4   5   6   >