I'll try again and look for logged errors. The problem I saw with
kafka-console-producer.sh did occur with a mixed version cluster,
so that is likely the problem. I'll try again with the 0.8.2.1 version.
Thanks,
Dave
On 1/19/2016 2:50 AM, Ismael Juma wrote:
Hi Dave,
Do you get any errors logged in the broker when you get ACK error 2
(InvalidMessage) while producing requests to a mixed version cluster? It
would be helpful to see them.
With regards to the kafka-console-producer.sh error, did you use the
0.9.0.0 console producer with a mixed version cluster (ie some brokers were
on 0.8.2.1 while others were on 0.9.0.0)? If so, it is expected that it
won't work correctly. All the brokers should be upgraded before the clients
are upgraded (otherwise the 0.8.2.1 broker will send a response that the
newer clients cannot handle).
Ismael
On Fri, Jan 15, 2016 at 7:52 PM, Dave Peterson <[email protected]> wrote:
Hi Ismael,
I'm using bruce (https://github.com/ifwe/bruce) to send the produce
requests, with a RequiredAcks value of 1. Everything works fine when
all brokers are running 0.8.2.1. Also if I set up a new 0.9.0.0
cluster from scratch rather than trying to upgrade, everything works
fine. The problem only occurs after upgrading one broker in the
3-broker cluster.
The topic I am sending to has 8 partitions numbered 0-7. Doing
further experimentation I see that the ACK error 2 occurs only when
I send to partition 7. No problems occur when sending to partitions
0-6. If it helps I can send output from "kafka-topics.sh --describe"
as well as tcpdump output showing the produce requests and responses.
For comparison I tried using the 0.9.0.0 version of
kafka-console-producer.sh to send messages. With the default
RequiredAcks value of 0, it worked although I don't know which
partition it sent to. With a RequiredAcks value of 1 I get the
output shown below.
Thanks,
Dave
[2016-01-15 10:33:28,843] ERROR Uncaught error in kafka producer I/O
thread: (org.apache.kafka.clients.producer.internals.Sender)
org.apache.kafka.common.protocol.types.SchemaException: Error reading field
'throttle_time_ms': java.nio.BufferUnderflowException
at
org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:464)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Thread.java:745)
On Fri, Jan 15, 2016 at 1:06 AM, Ismael Juma <[email protected]> wrote:
Hi Dave,
On Fri, Jan 15, 2016 at 2:04 AM, Dave Peterson <[email protected]>
wrote:
I was trying to upgrade an 0.8.2.1 broker cluster to 0.9.0.0 by
following
the instructions here:
http://kafka.apache.org/documentation.html#upgrade
After upgrading one broker, with inter.broker.protocol.version=0.8.2.X
set, I get ACK error 2 (InvalidMessage) when I try to send produce
requests.
I haven't seen other reports of this issue yet. Also, we have a system
test
that covers this scenario:
https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/upgrade_test.py
Just to double-check, what is the version of the producer that you are
using to send produce requests to the 0.9.0.0 broker when you get the
error?
Ismael
