Consumer group record fetch rate

2018-04-22 Thread Jamie Park 
Hi 

I am puzzled on this consumer group behavior.   I have 40 consumers within a 
single consumer group with large lags.  We are looking to increase partitions 
and number of consumers as lags indicate our records are produced faster than 
this consumer group can consume.  

However I observed this interesting behavior.  Hour 10, this consumer group 
consumed/produced 380 million records.  Using the same binary at hour 22, it 
produced 250 million records to its output topics.   Both hours contain 
considerable amount of lags.  Stream engine seems to be fetching records based 
on the rate that producers produce records.  If it’s capable of processing 380 
million records, why doesn’t  it fetch more records at hour 22 to get rid of 
lags?   I tried a few tweaks on consumer configuration to no avail.  Based on 
the metric we have on “received counts” we are definite not receiving enough 
records to address the lag.  Could anyone help/comment on this? 

Thank you 




Sent from my mobile device.

[jira] [Created] (KAFKA-6815) "default.production.exception.handler" default value is not specified correctly in KafkaStream doc

2018-04-22 Thread taekyung kim (JIRA) 
taekyung kim created KAFKA-6815:
---

 Summary: "default.production.exception.handler" default value is 
not specified correctly in KafkaStream doc
 Key: KAFKA-6815
 URL: https://issues.apache.org/jira/browse/KAFKA-6815
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Affects Versions: 1.1.0
Reporter: taekyung kim


 

[https://kafka.apache.org/11/documentation/streams/developer-guide/config-streams.html]

ver 1.1 document show invalid value
|default.deserialization.exception.handler|Medium|Exception handling class that 
implements the {{DeserializationExceptionHandler}}interface.|3 milliseconds|

 

actually it is  LogAndFailExceptionHandler class



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6814) Bad exception message for GroupIdNotFoundException/GroupNotEmptyException

2018-04-22 Thread huxihx (JIRA) 
huxihx created KAFKA-6814:
-

 Summary: Bad exception message for 
GroupIdNotFoundException/GroupNotEmptyException
 Key: KAFKA-6814
 URL: https://issues.apache.org/jira/browse/KAFKA-6814
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.1.0
Reporter: huxihx


Both `GroupNotEmptyException` and `GroupIdNotFoundException` constructors 
accept group id to deliver the exception message. However they can only be 
created with a fixed string "The group id does not exist" or "The group is not 
empty" which leads to a very unfriendly message when user failed to delete 
consumer groups, as below:
{code:java}
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.GroupIdNotFoundException: The group id The group 
id does not exist was not found{code}
group id got failed to be shown in the thrown exception. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)