Hi Grant, Yes, I have read and followed the Upgrade documentation and I have set inter.broker.protocol.version=0.8.2.X and it does not resolve the issue - based on the part of the code it's occurring in, it uses the 'latest' version for any API Key, and not the one specified by 'inter.broker.protocol.version'.
Matt -----Original Message----- From: Grant Henke [mailto:ghe...@cloudera.com] Sent: Thursday, November 5, 2015 3:28 PM To: dev@kafka.apache.org Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk) Hi Matthew, I have not read into the details of your issues but have done similar "rolling" upgrade testing myself. The reason replication breaks is due to some wire protocol changes. Just checking some preliminary things before digging in - Have you followed the upgrade steps outlined here? - https://kafka.apache.org/090/documentation.html#upgrade - Does setting inter.broker.protocol.version=0.8.2.X resolve the issue? - Note: you need to unset and restart again after all brokers are upgraded. In the future KIP-35 may help alleviate the manual step of setting the inter.broker.protocol.version. You can read more about KIP-35 and participate in the discussion/design here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version Thanks, Grant On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com> wrote: > Hello Kafka Devs, > > I've been testing the upgrade procedure between Kafka 0.8.2.1 and > Kafka > 0.9.0.0 and have been having Replication issues between the two > version, and I was wondering if anyone was aware of this issue (I just > searched and this seems to be related to KAFKA-2750 raised yesterday ). > > I start with 3 brokers running 0.8.2.1 all that contain data (1 topic > with > 10 partitions), then I shut down one of the brokers, upgrade it to > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in > broker.properties). Once the Broker is started I see errors like the > following: > > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600], > Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858<mailto: > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible cause: > org.apache.kafka.common.protocol.types.SchemaException: Error reading > field > 'responses': Error reading field 'topic': > java.nio.BufferUnderflowException > (kafka.server.ReplicaFetcherThread) > And > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600], > Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@224388b2<mailto: > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible cause: > org.apache.kafka.common.protocol.types.SchemaException: Error reading > field > 'responses': Error reading field 'partition_responses': Error reading > field > 'record_set': java.lang.IllegalArgumentException > (kafka.server.ReplicaFetcherThread) > > > I've spent some time in the Kafka code, and packet captures/wireshark > trying to figure this out, and I believe there is an issue in > org.apache.kafka.clients.networkClient.java in the > handleCompletedReceives > function: > When extracting the response body, this function is using > ProtoUtils.currentResponseSchema instead of ProtoUtils.ResponseSchema > and specifying the API version required by inter.broker.protocol.version. > Struct body = (Struct) > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload()); > > This results in errors when the newer version of a Schema > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against > the fetch response returned by the 0.8.2.1 broker > > > Thanks, > Matthew Bruce > mbr...@blackberry.com<mailto:mbr...@blackberry.com> > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
