Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Stevo Slavić]

Those docs at https://kafka.apache.org/090/documentation.html#upgrade ate
still mentioning 0.8.3 instead of 0.9.0. Is there JIRA already to fix this?

On Thu, Nov 5, 2015 at 9:28 PM, Grant Henke  wrote:

> Hi Matthew,
>
> I have not read into the details of your issues but have done similar
> "rolling" upgrade testing myself. The reason replication breaks is due to
> some wire protocol changes.
>
> Just checking some preliminary things before digging in
>
>- Have you followed the upgrade steps outlined here?
>   - https://kafka.apache.org/090/documentation.html#upgrade
>- Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
>   - Note: you need to unset and restart again after all brokers are
>   upgraded.
>
> In the future KIP-35 may help alleviate the manual step of setting the
> inter.broker.protocol.version. You can read more about KIP-35 and
> participate in the discussion/design here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>
> Thanks,
> Grant
>
>
> On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce 
> wrote:
>
> > Hello Kafka Devs,
> >
> > I've been testing the upgrade procedure between Kafka 0.8.2.1 and Kafka
> > 0.9.0.0 and have been having Replication issues between the two version,
> > and I was wondering if anyone was aware of this issue (I just searched
> and
> > this seems to be related to KAFKA-2750 raised yesterday ).
> >
> > I start with 3 brokers running 0.8.2.1 all that contain data (1 topic
> with
> > 10 partitions), then I shut down one of the brokers, upgrade it to 0.9.0
> > (making sure to set 'inter.broker.protocol.version=0.8.2.X' in
> > broker.properties).  Once the Broker is started I see errors like the
> > following:
> >
> > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600], Error
> > in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858 > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> field
> > 'responses': Error reading field 'topic':
> java.nio.BufferUnderflowException
> > (kafka.server.ReplicaFetcherThread)
> > And
> > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600], Error
> > in fetch kafka.server.ReplicaFetcherThread$FetchRequest@224388b2 > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> field
> > 'responses': Error reading field 'partition_responses': Error reading
> field
> > 'record_set': java.lang.IllegalArgumentException
> > (kafka.server.ReplicaFetcherThread)
> >
> >
> > I've spent some time in the Kafka code, and packet captures/wireshark
> > trying to figure this out, and I believe there is an issue in
> > org.apache.kafka.clients.networkClient.java in the
> handleCompletedReceives
> > function:
> > When extracting the response body, this function is using
> > ProtoUtils.currentResponseSchema instead of ProtoUtils.ResponseSchema
> and
> > specifying the API version required by inter.broker.protocol.version.
> > Struct body = (Struct)
> > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
> >
> > This results in errors when the newer version of a Schema
> > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against the
> > fetch response returned by the 0.8.2.1 broker
> >
> >
> > Thanks,
> > Matthew Bruce
> > mbr...@blackberry.com
> >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Guozhang Wang]

There is already a patch merged in trunk to change the version numbers, we
have not upload it to the kafka-site yet. It will be updated upon the 0.9.0
release.

Guozhang

On Fri, Nov 6, 2015 at 12:25 AM, Stevo Slavić  wrote:

> Those docs at https://kafka.apache.org/090/documentation.html#upgrade ate
> still mentioning 0.8.3 instead of 0.9.0. Is there JIRA already to fix this?
>
> On Thu, Nov 5, 2015 at 9:28 PM, Grant Henke  wrote:
>
> > Hi Matthew,
> >
> > I have not read into the details of your issues but have done similar
> > "rolling" upgrade testing myself. The reason replication breaks is due to
> > some wire protocol changes.
> >
> > Just checking some preliminary things before digging in
> >
> >- Have you followed the upgrade steps outlined here?
> >   - https://kafka.apache.org/090/documentation.html#upgrade
> >- Does setting inter.broker.protocol.version=0.8.2.X resolve the
> issue?
> >   - Note: you need to unset and restart again after all brokers are
> >   upgraded.
> >
> > In the future KIP-35 may help alleviate the manual step of setting the
> > inter.broker.protocol.version. You can read more about KIP-35 and
> > participate in the discussion/design here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > Thanks,
> > Grant
> >
> >
> > On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce 
> > wrote:
> >
> > > Hello Kafka Devs,
> > >
> > > I've been testing the upgrade procedure between Kafka 0.8.2.1 and Kafka
> > > 0.9.0.0 and have been having Replication issues between the two
> version,
> > > and I was wondering if anyone was aware of this issue (I just searched
> > and
> > > this seems to be related to KAFKA-2750 raised yesterday ).
> > >
> > > I start with 3 brokers running 0.8.2.1 all that contain data (1 topic
> > with
> > > 10 partitions), then I shut down one of the brokers, upgrade it to
> 0.9.0
> > > (making sure to set 'inter.broker.protocol.version=0.8.2.X' in
> > > broker.properties).  Once the Broker is started I see errors like the
> > > following:
> > >
> > > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> Error
> > > in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
>  > > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> > cause:
> > > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> > field
> > > 'responses': Error reading field 'topic':
> > java.nio.BufferUnderflowException
> > > (kafka.server.ReplicaFetcherThread)
> > > And
> > > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> Error
> > > in fetch kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
>  > > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> > cause:
> > > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> > field
> > > 'responses': Error reading field 'partition_responses': Error reading
> > field
> > > 'record_set': java.lang.IllegalArgumentException
> > > (kafka.server.ReplicaFetcherThread)
> > >
> > >
> > > I've spent some time in the Kafka code, and packet captures/wireshark
> > > trying to figure this out, and I believe there is an issue in
> > > org.apache.kafka.clients.networkClient.java in the
> > handleCompletedReceives
> > > function:
> > > When extracting the response body, this function is using
> > > ProtoUtils.currentResponseSchema instead of ProtoUtils.ResponseSchema
> > and
> > > specifying the API version required by inter.broker.protocol.version.
> > > Struct body = (Struct)
> > > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
> > >
> > > This results in errors when the newer version of a Schema
> > > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against the
> > > fetch response returned by the 0.8.2.1 broker
> > >
> > >
> > > Thanks,
> > > Matthew Bruce
> > > mbr...@blackberry.com
> > >
> > >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>



-- 
-- Guozhang

RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Matthew Bruce]

Hi Grant,

Creating a patch for this issue actually wasn't a whole lot of work.  I've 
uploaded one to the JIRA ticket.

Thanks,
Matt

-Original Message-
From: Matthew Bruce 
Sent: Thursday, November 5, 2015 3:52 PM
To: 'dev@kafka.apache.org' <dev@kafka.apache.org>
Subject: RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

https://issues.apache.org/jira/browse/KAFKA-2756

-Original Message-
From: Grant Henke [mailto:ghe...@cloudera.com]
Sent: Thursday, November 5, 2015 3:37 PM
To: dev@kafka.apache.org
Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

Hi Matt,

Great. Just wanted to be sure of the background. Are you interested in opening 
a jira and contributing a patch? If not would you be interested in testing a 
patch if I provided one?

Thank you,
Grant

On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hi Grant,
>
> Yes, I have read and followed the Upgrade documentation and I have set 
> inter.broker.protocol.version=0.8.2.X and it does not resolve the 
> issue - based on the part of the code it's occurring in, it uses the 'latest'
> version for any API Key, and not the one specified by 
> 'inter.broker.protocol.version'.
>
> Matt
>
> -Original Message-
> From: Grant Henke [mailto:ghe...@cloudera.com]
> Sent: Thursday, November 5, 2015 3:28 PM
> To: dev@kafka.apache.org
> Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> Hi Matthew,
>
> I have not read into the details of your issues but have done similar 
> "rolling" upgrade testing myself. The reason replication breaks is due 
> to some wire protocol changes.
>
> Just checking some preliminary things before digging in
>
>- Have you followed the upgrade steps outlined here?
>   - https://kafka.apache.org/090/documentation.html#upgrade
>- Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
>   - Note: you need to unset and restart again after all brokers are
>   upgraded.
>
> In the future KIP-35 may help alleviate the manual step of setting the 
> inter.broker.protocol.version. You can read more about KIP-35 and 
> participate in the discussion/design here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+
> protocol+version
>
> Thanks,
> Grant
>
>
> On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com>
> wrote:
>
> > Hello Kafka Devs,
> >
> > I've been testing the upgrade procedure between Kafka 0.8.2.1 and 
> > Kafka
> > 0.9.0.0 and have been having Replication issues between the two 
> > version, and I was wondering if anyone was aware of this issue (I 
> > just searched and this seems to be related to KAFKA-2750 raised yesterday ).
> >
> > I start with 3 brokers running 0.8.2.1 all that contain data (1 
> > topic with
> > 10 partitions), then I shut down one of the brokers, upgrade it to
> > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in 
> > broker.properties).  Once the Broker is started I see errors like 
> > the
> > following:
> >
> > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> > Error in fetch
> > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
>  > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'topic':
> > java.nio.BufferUnderflowException
> > (kafka.server.ReplicaFetcherThread)
> > And
> > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> > Error in fetch
> > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
>  > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'partition_responses': Error 
> > reading field
> > 'record_set': java.lang.IllegalArgumentException
> > (kafka.server.ReplicaFetcherThread)
> >
> >
> > I've spent some time in the Kafka code, and packet 
> > captures/wireshark trying to figure this out, and I believe there is 
> > an issue in org.apache.kafka.clients.networkClient.java in the 
> > handleCompletedReceives
> > function:
> > When extracting the response body, this function is using 
> > ProtoUtils.currentResponseSchema instead of 
> > ProtoUtils.ResponseSchema and specifying the API version required by 
> > inter.broker.protocol.version.
> > Struct body = (Struct) 
&

Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Grant Henke]

Hi Matt,

Great. Just wanted to be sure of the background. Are you interested in
opening a jira and contributing a patch? If not would you be interested in
testing a patch if I provided one?

Thank you,
Grant

On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hi Grant,
>
> Yes, I have read and followed the Upgrade documentation and I have set
> inter.broker.protocol.version=0.8.2.X and it does not resolve the issue -
> based on the part of the code it's occurring in, it uses the 'latest'
> version for any API Key, and not the one specified by
> 'inter.broker.protocol.version'.
>
> Matt
>
> -Original Message-
> From: Grant Henke [mailto:ghe...@cloudera.com]
> Sent: Thursday, November 5, 2015 3:28 PM
> To: dev@kafka.apache.org
> Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> Hi Matthew,
>
> I have not read into the details of your issues but have done similar
> "rolling" upgrade testing myself. The reason replication breaks is due to
> some wire protocol changes.
>
> Just checking some preliminary things before digging in
>
>- Have you followed the upgrade steps outlined here?
>   - https://kafka.apache.org/090/documentation.html#upgrade
>- Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
>   - Note: you need to unset and restart again after all brokers are
>   upgraded.
>
> In the future KIP-35 may help alleviate the manual step of setting the
> inter.broker.protocol.version. You can read more about KIP-35 and
> participate in the discussion/design here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>
> Thanks,
> Grant
>
>
> On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com>
> wrote:
>
> > Hello Kafka Devs,
> >
> > I've been testing the upgrade procedure between Kafka 0.8.2.1 and
> > Kafka
> > 0.9.0.0 and have been having Replication issues between the two
> > version, and I was wondering if anyone was aware of this issue (I just
> > searched and this seems to be related to KAFKA-2750 raised yesterday ).
> >
> > I start with 3 brokers running 0.8.2.1 all that contain data (1 topic
> > with
> > 10 partitions), then I shut down one of the brokers, upgrade it to
> > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in
> > broker.properties).  Once the Broker is started I see errors like the
> > following:
> >
> > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> > Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
>  > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> > field
> > 'responses': Error reading field 'topic':
> > java.nio.BufferUnderflowException
> > (kafka.server.ReplicaFetcherThread)
> > And
> > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> > Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
>  > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error reading
> > field
> > 'responses': Error reading field 'partition_responses': Error reading
> > field
> > 'record_set': java.lang.IllegalArgumentException
> > (kafka.server.ReplicaFetcherThread)
> >
> >
> > I've spent some time in the Kafka code, and packet captures/wireshark
> > trying to figure this out, and I believe there is an issue in
> > org.apache.kafka.clients.networkClient.java in the
> > handleCompletedReceives
> > function:
> > When extracting the response body, this function is using
> > ProtoUtils.currentResponseSchema instead of ProtoUtils.ResponseSchema
> > and specifying the API version required by inter.broker.protocol.version.
> > Struct body = (Struct)
> > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
> >
> > This results in errors when the newer version of a Schema
> > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against
> > the fetch response returned by the 0.8.2.1 broker
> >
> >
> > Thanks,
> > Matthew Bruce
> > mbr...@blackberry.com<mailto:mbr...@blackberry.com>
> >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Matthew Bruce]

Hi Grant,

Yes, I have read and followed the Upgrade documentation and I have set 
inter.broker.protocol.version=0.8.2.X and it does not resolve the issue - based 
on the part of the code it's occurring in, it uses the 'latest' version for any 
API Key, and not the one specified by  'inter.broker.protocol.version'.

Matt

-Original Message-
From: Grant Henke [mailto:ghe...@cloudera.com] 
Sent: Thursday, November 5, 2015 3:28 PM
To: dev@kafka.apache.org
Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

Hi Matthew,

I have not read into the details of your issues but have done similar "rolling" 
upgrade testing myself. The reason replication breaks is due to some wire 
protocol changes.

Just checking some preliminary things before digging in

   - Have you followed the upgrade steps outlined here?
  - https://kafka.apache.org/090/documentation.html#upgrade
   - Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
  - Note: you need to unset and restart again after all brokers are
  upgraded.

In the future KIP-35 may help alleviate the manual step of setting the 
inter.broker.protocol.version. You can read more about KIP-35 and participate 
in the discussion/design here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version

Thanks,
Grant


On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hello Kafka Devs,
>
> I've been testing the upgrade procedure between Kafka 0.8.2.1 and 
> Kafka
> 0.9.0.0 and have been having Replication issues between the two 
> version, and I was wondering if anyone was aware of this issue (I just 
> searched and this seems to be related to KAFKA-2750 raised yesterday ).
>
> I start with 3 brokers running 0.8.2.1 all that contain data (1 topic 
> with
> 10 partitions), then I shut down one of the brokers, upgrade it to 
> 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in 
> broker.properties).  Once the Broker is started I see errors like the
> following:
>
> [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600], 
> Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858 kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible cause:
> org.apache.kafka.common.protocol.types.SchemaException: Error reading 
> field
> 'responses': Error reading field 'topic': 
> java.nio.BufferUnderflowException
> (kafka.server.ReplicaFetcherThread)
> And
> [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600], 
> Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@224388b2 kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible cause:
> org.apache.kafka.common.protocol.types.SchemaException: Error reading 
> field
> 'responses': Error reading field 'partition_responses': Error reading 
> field
> 'record_set': java.lang.IllegalArgumentException
> (kafka.server.ReplicaFetcherThread)
>
>
> I've spent some time in the Kafka code, and packet captures/wireshark 
> trying to figure this out, and I believe there is an issue in 
> org.apache.kafka.clients.networkClient.java in the 
> handleCompletedReceives
> function:
> When extracting the response body, this function is using 
> ProtoUtils.currentResponseSchema instead of ProtoUtils.ResponseSchema  
> and specifying the API version required by inter.broker.protocol.version.
> Struct body = (Struct) 
> ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
>
> This results in errors when the newer version of a Schema
> (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against 
> the fetch response returned by the 0.8.2.1 broker
>
>
> Thanks,
> Matthew Bruce
> mbr...@blackberry.com<mailto:mbr...@blackberry.com>
>
>


--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Matthew Bruce]

I don’t mind opening a JIRA and testing a patch, I'm not sure how quickly I'd 
be able to contribute one on my own though.

Thanks,
Matt

-Original Message-
From: Grant Henke [mailto:ghe...@cloudera.com] 
Sent: Thursday, November 5, 2015 3:37 PM
To: dev@kafka.apache.org
Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

Hi Matt,

Great. Just wanted to be sure of the background. Are you interested in opening 
a jira and contributing a patch? If not would you be interested in testing a 
patch if I provided one?

Thank you,
Grant

On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hi Grant,
>
> Yes, I have read and followed the Upgrade documentation and I have set 
> inter.broker.protocol.version=0.8.2.X and it does not resolve the 
> issue - based on the part of the code it's occurring in, it uses the 'latest'
> version for any API Key, and not the one specified by 
> 'inter.broker.protocol.version'.
>
> Matt
>
> -Original Message-
> From: Grant Henke [mailto:ghe...@cloudera.com]
> Sent: Thursday, November 5, 2015 3:28 PM
> To: dev@kafka.apache.org
> Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> Hi Matthew,
>
> I have not read into the details of your issues but have done similar 
> "rolling" upgrade testing myself. The reason replication breaks is due 
> to some wire protocol changes.
>
> Just checking some preliminary things before digging in
>
>- Have you followed the upgrade steps outlined here?
>   - https://kafka.apache.org/090/documentation.html#upgrade
>- Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
>   - Note: you need to unset and restart again after all brokers are
>   upgraded.
>
> In the future KIP-35 may help alleviate the manual step of setting the 
> inter.broker.protocol.version. You can read more about KIP-35 and 
> participate in the discussion/design here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+
> protocol+version
>
> Thanks,
> Grant
>
>
> On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com>
> wrote:
>
> > Hello Kafka Devs,
> >
> > I've been testing the upgrade procedure between Kafka 0.8.2.1 and 
> > Kafka
> > 0.9.0.0 and have been having Replication issues between the two 
> > version, and I was wondering if anyone was aware of this issue (I 
> > just searched and this seems to be related to KAFKA-2750 raised yesterday ).
> >
> > I start with 3 brokers running 0.8.2.1 all that contain data (1 
> > topic with
> > 10 partitions), then I shut down one of the brokers, upgrade it to
> > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in 
> > broker.properties).  Once the Broker is started I see errors like 
> > the
> > following:
> >
> > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> > Error in fetch 
> > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
>  > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'topic':
> > java.nio.BufferUnderflowException
> > (kafka.server.ReplicaFetcherThread)
> > And
> > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> > Error in fetch 
> > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
>  > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'partition_responses': Error 
> > reading field
> > 'record_set': java.lang.IllegalArgumentException
> > (kafka.server.ReplicaFetcherThread)
> >
> >
> > I've spent some time in the Kafka code, and packet 
> > captures/wireshark trying to figure this out, and I believe there is 
> > an issue in org.apache.kafka.clients.networkClient.java in the 
> > handleCompletedReceives
> > function:
> > When extracting the response body, this function is using 
> > ProtoUtils.currentResponseSchema instead of 
> > ProtoUtils.ResponseSchema and specifying the API version required by 
> > inter.broker.protocol.version.
> > Struct body = (Struct) 
> > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
> >
> > This results in errors when the newer version of a Schema
> > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against 
> > the fetch response returned by the 0.8.2.1 broker
> >
> >
> > Thanks,
> > Matthew Bruce
> > mbr...@blackberry.com<mailto:mbr...@blackberry.com>
> >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Matthew Bruce]

https://issues.apache.org/jira/browse/KAFKA-2756

-Original Message-
From: Grant Henke [mailto:ghe...@cloudera.com] 
Sent: Thursday, November 5, 2015 3:37 PM
To: dev@kafka.apache.org
Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

Hi Matt,

Great. Just wanted to be sure of the background. Are you interested in opening 
a jira and contributing a patch? If not would you be interested in testing a 
patch if I provided one?

Thank you,
Grant

On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hi Grant,
>
> Yes, I have read and followed the Upgrade documentation and I have set 
> inter.broker.protocol.version=0.8.2.X and it does not resolve the 
> issue - based on the part of the code it's occurring in, it uses the 'latest'
> version for any API Key, and not the one specified by 
> 'inter.broker.protocol.version'.
>
> Matt
>
> -Original Message-
> From: Grant Henke [mailto:ghe...@cloudera.com]
> Sent: Thursday, November 5, 2015 3:28 PM
> To: dev@kafka.apache.org
> Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> Hi Matthew,
>
> I have not read into the details of your issues but have done similar 
> "rolling" upgrade testing myself. The reason replication breaks is due 
> to some wire protocol changes.
>
> Just checking some preliminary things before digging in
>
>- Have you followed the upgrade steps outlined here?
>   - https://kafka.apache.org/090/documentation.html#upgrade
>- Does setting inter.broker.protocol.version=0.8.2.X resolve the issue?
>   - Note: you need to unset and restart again after all brokers are
>   upgraded.
>
> In the future KIP-35 may help alleviate the manual step of setting the 
> inter.broker.protocol.version. You can read more about KIP-35 and 
> participate in the discussion/design here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+
> protocol+version
>
> Thanks,
> Grant
>
>
> On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com>
> wrote:
>
> > Hello Kafka Devs,
> >
> > I've been testing the upgrade procedure between Kafka 0.8.2.1 and 
> > Kafka
> > 0.9.0.0 and have been having Replication issues between the two 
> > version, and I was wondering if anyone was aware of this issue (I 
> > just searched and this seems to be related to KAFKA-2750 raised yesterday ).
> >
> > I start with 3 brokers running 0.8.2.1 all that contain data (1 
> > topic with
> > 10 partitions), then I shut down one of the brokers, upgrade it to
> > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in 
> > broker.properties).  Once the Broker is started I see errors like 
> > the
> > following:
> >
> > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> > Error in fetch 
> > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
>  > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'topic':
> > java.nio.BufferUnderflowException
> > (kafka.server.ReplicaFetcherThread)
> > And
> > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> > Error in fetch 
> > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
>  > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> cause:
> > org.apache.kafka.common.protocol.types.SchemaException: Error 
> > reading field
> > 'responses': Error reading field 'partition_responses': Error 
> > reading field
> > 'record_set': java.lang.IllegalArgumentException
> > (kafka.server.ReplicaFetcherThread)
> >
> >
> > I've spent some time in the Kafka code, and packet 
> > captures/wireshark trying to figure this out, and I believe there is 
> > an issue in org.apache.kafka.clients.networkClient.java in the 
> > handleCompletedReceives
> > function:
> > When extracting the response body, this function is using 
> > ProtoUtils.currentResponseSchema instead of 
> > ProtoUtils.ResponseSchema and specifying the API version required by 
> > inter.broker.protocol.version.
> > Struct body = (Struct) 
> > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload());
> >
> > This results in errors when the newer version of a Schema
> > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against 
> > the fetch response returned by the 0.8.2.1 broker
> >
> >
> > Thanks,
> > Matthew Bruce
> > mbr...@blackberry.com<mailto:mbr...@blackberry.com>
> >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)

[Grant Henke]

Thanks for the contribution Matt!

On Thu, Nov 5, 2015 at 4:21 PM, Matthew Bruce <mbr...@blackberry.com> wrote:

> Hi Grant,
>
> Creating a patch for this issue actually wasn't a whole lot of work.  I've
> uploaded one to the JIRA ticket.
>
> Thanks,
> Matt
>
> -Original Message-
> From: Matthew Bruce
> Sent: Thursday, November 5, 2015 3:52 PM
> To: 'dev@kafka.apache.org' <dev@kafka.apache.org>
> Subject: RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> https://issues.apache.org/jira/browse/KAFKA-2756
>
> -Original Message-
> From: Grant Henke [mailto:ghe...@cloudera.com]
> Sent: Thursday, November 5, 2015 3:37 PM
> To: dev@kafka.apache.org
> Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
>
> Hi Matt,
>
> Great. Just wanted to be sure of the background. Are you interested in
> opening a jira and contributing a patch? If not would you be interested in
> testing a patch if I provided one?
>
> Thank you,
> Grant
>
> On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com>
> wrote:
>
> > Hi Grant,
> >
> > Yes, I have read and followed the Upgrade documentation and I have set
> > inter.broker.protocol.version=0.8.2.X and it does not resolve the
> > issue - based on the part of the code it's occurring in, it uses the
> 'latest'
> > version for any API Key, and not the one specified by
> > 'inter.broker.protocol.version'.
> >
> > Matt
> >
> > -Original Message-
> > From: Grant Henke [mailto:ghe...@cloudera.com]
> > Sent: Thursday, November 5, 2015 3:28 PM
> > To: dev@kafka.apache.org
> > Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk)
> >
> > Hi Matthew,
> >
> > I have not read into the details of your issues but have done similar
> > "rolling" upgrade testing myself. The reason replication breaks is due
> > to some wire protocol changes.
> >
> > Just checking some preliminary things before digging in
> >
> >- Have you followed the upgrade steps outlined here?
> >   - https://kafka.apache.org/090/documentation.html#upgrade
> >- Does setting inter.broker.protocol.version=0.8.2.X resolve the
> issue?
> >   - Note: you need to unset and restart again after all brokers are
> >   upgraded.
> >
> > In the future KIP-35 may help alleviate the manual step of setting the
> > inter.broker.protocol.version. You can read more about KIP-35 and
> > participate in the discussion/design here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+
> > protocol+version
> >
> > Thanks,
> > Grant
> >
> >
> > On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com>
> > wrote:
> >
> > > Hello Kafka Devs,
> > >
> > > I've been testing the upgrade procedure between Kafka 0.8.2.1 and
> > > Kafka
> > > 0.9.0.0 and have been having Replication issues between the two
> > > version, and I was wondering if anyone was aware of this issue (I
> > > just searched and this seems to be related to KAFKA-2750 raised
> yesterday ).
> > >
> > > I start with 3 brokers running 0.8.2.1 all that contain data (1
> > > topic with
> > > 10 partitions), then I shut down one of the brokers, upgrade it to
> > > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in
> > > broker.properties).  Once the Broker is started I see errors like
> > > the
> > > following:
> > >
> > > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600],
> > > Error in fetch
> > > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858
> >  > > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible
> > cause:
> > > org.apache.kafka.common.protocol.types.SchemaException: Error
> > > reading field
> > > 'responses': Error reading field 'topic':
> > > java.nio.BufferUnderflowException
> > > (kafka.server.ReplicaFetcherThread)
> > > And
> > > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600],
> > > Error in fetch
> > > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2
> >  > > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible
> > cause:
> > > org.apache.kafka.common.protocol.types.SchemaException: Error
> > > reading field
> > > 'responses': Error reading field 'partition_responses': Error
> > > reading field
> > > 'record_set':