[jira] [Commented] (KAFKA-1749) Brokers continually throw exceptions when there are hundreds of topic being fetched by mirrormaker

2014-11-07 Thread Manikumar Reddy (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201983#comment-14201983
 ] 

Manikumar Reddy commented on KAFKA-1749:


Created reviewboard https://reviews.apache.org/r/27722/diff/
 against branch origin

> Brokers continually throw exceptions when there are hundreds of topic being 
> fetched by mirrormaker
> --
>
> Key: KAFKA-1749
> URL: https://issues.apache.org/jira/browse/KAFKA-1749
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: Min Zhou
> Attachments: KAFKA-1749.patch
>
>
> Here is one piece of  millions of the exceptions. 
> {noformat}
> kafka.common.KafkaException: This operation cannot be completed on a complete 
> request.
> at kafka.network.Transmission$class.expectIncomplete(Transmission.scala:34)
> at kafka.api.FetchResponseSend.expectIncomplete(FetchResponse.scala:191)
> at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:214)
> at kafka.network.Processor.write(SocketServer.scala:375)
> at kafka.network.Processor.run(SocketServer.scala:247)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> We use tools to hook function kafka.api.FetchResponseSend.writeTo, found 
>  fetchResponse.sizeInBytes was overflow.  Which means below code will get a 
> result over the limit of integer type
> {noformat}
> val sizeInBytes =
> FetchResponse.headerSize +
> dataGroupedByTopic.foldLeft(0) ((folded, curr) => {
>   val topicData = TopicData(curr._1, curr._2.map {
> case (topicAndPartition, partitionData) => 
> (topicAndPartition.partition, partitionData)
>   })
>   folded + topicData.sizeInBytes
> })
> {noformat} 
>  If we just fetch few topics by mirrormaker, the brokers ran well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1749) Brokers continually throw exceptions when there are hundreds of topic being fetched by mirrormaker

2014-11-03 Thread Ewen Cheslack-Postava (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14195417#comment-14195417
 ] 

Ewen Cheslack-Postava commented on KAFKA-1749:
--

This looks like it's probably a duplicate of KAFKA-1196 -- that looks like the 
stack trace I described in that issue that I get when sending a FetchResponse 
that exceeds 2GB.

> Brokers continually throw exceptions when there are hundreds of topic being 
> fetched by mirrormaker
> --
>
> Key: KAFKA-1749
> URL: https://issues.apache.org/jira/browse/KAFKA-1749
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1, 0.8.2
>Reporter: Min Zhou
>
> Here is one piece of  millions of the exceptions. 
> {noformat}
> kafka.common.KafkaException: This operation cannot be completed on a complete 
> request.
> at kafka.network.Transmission$class.expectIncomplete(Transmission.scala:34)
> at kafka.api.FetchResponseSend.expectIncomplete(FetchResponse.scala:191)
> at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:214)
> at kafka.network.Processor.write(SocketServer.scala:375)
> at kafka.network.Processor.run(SocketServer.scala:247)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> We use tools to hook function kafka.api.FetchResponseSend.writeTo, found 
>  fetchResponse.sizeInBytes was overflow.  Which means below code will get a 
> result over the limit of integer type
> {noformat}
> val sizeInBytes =
> FetchResponse.headerSize +
> dataGroupedByTopic.foldLeft(0) ((folded, curr) => {
>   val topicData = TopicData(curr._1, curr._2.map {
> case (topicAndPartition, partitionData) => 
> (topicAndPartition.partition, partitionData)
>   })
>   folded + topicData.sizeInBytes
> })
> {noformat} 
>  If we just fetch few topics by mirrormaker, the brokers ran well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)