[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201702#comment-14201702 ] Honghai Chen commented on KAFKA-391: Many thanks for you help. After debugging and testing, seemly I can't hit that exception. Actually we're using one c# version client which is inherit from https://github.com/precog/kafka/tree/master/clients/csharp/src/Kafka/Kafka.Client , and after debug and compare it's code with java version, finally prove that it's the bug of the C# code. In java version, when create ProducerRequest, it set produceRequest.data as messagesPerTopic, and do group by topic just before send binary. But in our c# version, it group it first and set the produceRequest.data as dictionary of Topic,Data, so we hit this exception wrongly, we fixed it. Many thanks for your time. But anyway, can't find our related open source version from internet, our version has DefaultCallbackHandler.cs, but the version on https://github.com/precog/kafka/tree/master/clients/csharp/src/Kafka/Kafka.Client has no, so can't provide the link here. The java link: https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=blob;f=core/src/main/scala/kafka/api/ProducerRequest.scala;h=570b2da1d865086f9830aa919a49063abbbe574d;hb=HEAD https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=blob;f=core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala;h=821901e4f434dfd9eec6eceabfc2e1e65507a57c;hb=HEAD#l260 Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198699#comment-14198699 ] Joel Koshy commented on KAFKA-391: -- If there are three partitions, then there will be three message-sets. i.e., producerRequest.data.size will be three, not one. Can you give example _application_ code that reproduces the issue that you are seeing? Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[
https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195762#comment-14195762
]
Honghai Chen commented on KAFKA-391:
This situation happen under below scenario:
one broker is leader for several partitions, for example 3, when send one
messageset which has message for all of the 3 partitions of this broker ,
the response.status.size is 3 and the producerRequest.data.size is 1.then
it hit this exception. Any idea for fix? Do we need compare
response.status.size with messagesPerTopic.Count instead of
producerRequest.data.size ?
private def send(brokerId: Int, messagesPerTopic:
collection.mutable.Map[TopicAndPartition, ByteBufferMessageSet]) = {
if(brokerId 0) {
warn(Failed to send data since partitions %s don't have a
leader.format(messagesPerTopic.map(_._1).mkString(,)))
messagesPerTopic.keys.toSeq
} else if(messagesPerTopic.size 0) {
val currentCorrelationId = correlationId.getAndIncrement
val producerRequest = new ProducerRequest(currentCorrelationId,
config.clientId, config.requestRequiredAcks,
config.requestTimeoutMs, messagesPerTopic)
var failedTopicPartitions = Seq.empty[TopicAndPartition]
try {
val syncProducer = producerPool.getProducer(brokerId)
debug(Producer sending messages with correlation id %d for topics %s
to broker %d on %s:%d
.format(currentCorrelationId, messagesPerTopic.keySet.mkString(,),
brokerId, syncProducer.config.host, syncProducer.config.port))
val response = syncProducer.send(producerRequest)
debug(Producer sent messages with correlation id %d for topics %s to
broker %d on %s:%d
.format(currentCorrelationId, messagesPerTopic.keySet.mkString(,),
brokerId, syncProducer.config.host, syncProducer.config.port))
if(response != null) {
if (response.status.size != producerRequest.data.size)
throw new KafkaException(Incomplete response (%s) for producer
request (%s).format(response, producerRequest))
Producer request and response classes should use maps
-
Key: KAFKA-391
URL: https://issues.apache.org/jira/browse/KAFKA-391
Project: Kafka
Issue Type: Bug
Reporter: Joel Koshy
Assignee: Joel Koshy
Priority: Blocker
Labels: optimization
Fix For: 0.8.0
Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch,
KAFKA-391-v3.patch, KAFKA-391-v4.patch
Producer response contains two arrays of error codes and offsets - the
ordering in these arrays correspond to the flattened ordering of the request
arrays.
It would be better to switch to maps in the request and response as this
would make the code clearer and more efficient (right now, linear scans are
used in handling producer acks).
We can probably do the same in the fetch request/response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181170#comment-14181170 ] Honghai Chen commented on KAFKA-391: Why this fix add this line code? Sometimes got 2 responses for one request, Why fall into this situation, will there be duplicate data in Kafka? https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commitdiff;h=b688c3ba045df340bc32caa40ba1909eddbcbec5 +if (response.status.size != producerRequest.data.size)+ throw new KafkaException(Incomplete response (%s) for producer request (%s)+ .format(response, producerRequest)) Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182057#comment-14182057 ] Jun Rao commented on KAFKA-391: --- The client should always receive one response per request. Are you see otherwise? Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182097#comment-14182097 ] Honghai Chen commented on KAFKA-391: Yes, after add more information to the error message , sometimes see two response for one request. Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182197#comment-14182197 ] Jun Rao commented on KAFKA-391: --- Which version of Kafka are you using? Is that easily reproducible? Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182240#comment-14182240 ] Joel Koshy commented on KAFKA-391: -- You mean two responses to the caller of send? I don't see why those two lines would cause two responses. Can you explain further and provide steps to reproduce if there really is an issue? Do you see errors/warns in the logs? Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-391) Producer request and response classes should use maps
[ https://issues.apache.org/jira/browse/KAFKA-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182276#comment-14182276 ] Joel Koshy commented on KAFKA-391: -- Yeah I see your point. That's interesting - I actually don't remember why that was added but it appears there must have been a legitimate reason (since you ran into it:) ). Since you are able to reproduce it can you actually print the full original request and response itself? It should be in the exception that is thrown. Also, what is your broker Kafka version? Also, what is the version of the producer? Is it the same? Producer request and response classes should use maps - Key: KAFKA-391 URL: https://issues.apache.org/jira/browse/KAFKA-391 Project: Kafka Issue Type: Bug Reporter: Joel Koshy Assignee: Joel Koshy Priority: Blocker Labels: optimization Fix For: 0.8.0 Attachments: KAFKA-391-draft-r1374069.patch, KAFKA-391-v2.patch, KAFKA-391-v3.patch, KAFKA-391-v4.patch Producer response contains two arrays of error codes and offsets - the ordering in these arrays correspond to the flattened ordering of the request arrays. It would be better to switch to maps in the request and response as this would make the code clearer and more efficient (right now, linear scans are used in handling producer acks). We can probably do the same in the fetch request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
