Re: kafka.consumers Java Package

[Chris Barlock]

Thank you!

Chris




From:   Eric Azama 
To: users@kafka.apache.org
Date:   11/09/2018 12:27 PM
Subject:Re: kafka.consumers Java Package



The kafka.consumer package was the old scala consumer. The upgrade notes
<
https://kafka.apache.org/documentation/#upgrade
> indicate that they were
deprecated in 0.11 and removed in 2.0


On Fri, Nov 9, 2018 at 8:00 AM Harsha Chintalapani  
wrote:

> Chris,
> You are upgrading from 0.10.2.2 to 2.0.0 . There will be quite 
few
> changes and it looks like you might be using classes other than
> KafkaConsumer which are not public API? What classes specifically are 
not
> available.
>
> -Harsha
> On Nov 9, 2018, 7:47 AM -0800, Chris Barlock , 
wrote:
> > I was trying to move some of our Kafka client code from
> > kafka_2.11_0.10.2.2 to kafka_2.11_2.0.0. However, we have some code 
that
> > uses code in the kafka.consumer package. That appears to be either 
moved
> > or removed from kafka_2.11_2.0.0. Is it gone or moved?
> >
> > Thanks!
> >
> > Chris
> >
>

kafka.consumers Java Package

[Chris Barlock]

I was trying to move some of our Kafka client code from 
kafka_2.11_0.10.2.2 to kafka_2.11_2.0.0.  However, we have some code that 
uses code in the kafka.consumer package.  That appears to be either moved 
or removed from kafka_2.11_2.0.0.  Is it gone or moved?

Thanks!

Chris

Expose Kafka Server Configuration

[Chris Barlock]

Does Kafka expose its server configuration in any way that I can get it 
programmatically?  Specifically, I'm interested in knowing the 
message.max.bytes value.

Chris

Re: TopicMetadata Question

[Chris Barlock]

Thank you Radek, but that is also not very interesting.  topic() returns 
just the topic name,   If there is no "easy" equivalent to getting the 0.8 
kafka.api.TopicMetadata, it looks like I could get some information from 
the PartitionMetadata that this TopicMetadata holds.

Chris

IBM Middleware
Research Triangle Park, NC
(919) 224-2240
Internet:  barl...@us.ibm.com



From:   Radoslaw Gruchalski <ra...@gruchalski.com>
To:     Chris Barlock/Raleigh/IBM@IBMUS, users@kafka.apache.org
Date:   07/26/2016 03:10 PM
Subject:Re: TopicMetadata Question



Chris,

There is a .topic() method available on that object:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java#L323


–
Best regards,
Radek Gruchalski
ra...@gruchalski.com
de.linkedin.com/in/radgruchalski


*Confidentiality:*This communication is intended for the above-named 
person
and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor
must you copy or show it to anyone; please delete/destroy and inform the
sender immediately.

On July 26, 2016 at 8:08:18 PM, Chris Barlock (barl...@us.ibm.com) wrote:

Starting with Kafka 0.8.2.1, we have some code that call
AdminUtils.fetchTopicMetadataFromZk. This returned a
kafka.api.TopicMetadata object which provided some useful information
about the topic. With 0.10.0.0, fetchTopicMetadataFromZk returns
org.apache.kafka.common.requests.MetadataResponse.TopicMetadata. This
TopicMetadata does not have a toString method, so we get the rather
useless version from Object.

Does Kafka 0.10 provide some other way to get topic metadata?

Chris

TopicMetadata Question

[Chris Barlock]

Starting with Kafka 0.8.2.1, we have some code that call 
AdminUtils.fetchTopicMetadataFromZk.  This returned a 
kafka.api.TopicMetadata object which provided some useful information 
about the topic.  With 0.10.0.0, fetchTopicMetadataFromZk returns 
org.apache.kafka.common.requests.MetadataResponse.TopicMetadata.  This 
TopicMetadata does not have a toString method, so we get the rather 
useless version from Object.

Does Kafka 0.10 provide some other way to get topic metadata?

Chris

Re: release of 0.10.1

[Chris Barlock]

So this begs the question -- why doesn't Kafka join the mainstream and 
have a 1.0 release?  The 0.x versioning makes one think that this code is, 
at best, beta.

Chris




From:   Ewen Cheslack-Postava 
To: "users@kafka.apache.org" 
Date:   07/23/2016 09:03 PM
Subject:Re: release of 0.10.1



0.10.1.0 is considered a major release. The release 0.10.0.0 might have a
follow up 0.10.0.1 for critical bug fixes, but 0.10.1.0 is a "minor"
release. Kafka is a bit odd in that its "major" releases are labeled as a
normal "minor" release number because Kafka hasn't decided to make an
official 1.0 release yet.

What features/fixes are you looking for?

-Ewen

On Wed, Jul 20, 2016 at 7:23 AM, David Garcia  
wrote:

> Does anyone know when this release will be cut?
>
> -David
>



-- 
Thanks,
Ewen

Compatibility question

[Chris Barlock]

Can the kafka-clients Maven bundle at version 0.8.2.1 be used to 
communicate with a Kafka 0.10.0.0 server?

Chris

Re: Producer Properties

[Chris Barlock]

Thank you, Dan!

Chris





From:   "Dan Bahir (BLOOMBERG/ 120 PARK)" <dba...@bloomberg.net>
To: users@kafka.apache.org
Date:   06/28/2016 09:49 AM
Subject:Re: Producer Properties



Hi Chris, 

The new producer returns a future so it works in an async manner hence no 
need for the producer.type property.

0.8 -> 0.10
batch.num.messages -> batch.size
queue.buffering.max.ms -> linger.ms

Your assumptions for the serializers look correct.

Take a look at the documentation for both versions.
http://kafka.apache.org/081/documentation.html and 
http://kafka.apache.org/081/documentation.html.

Hope that helps


From: users@kafka.apache.org At: 06/27/16 09:34:55
To: users@kafka.apache.org
Subject: Re: Producer Properties

Anybody?

Chris


From:   Chris Barlock/Raleigh/IBM@IBMUS
To: users@kafka.apache.org
Date:   06/24/2016 04:56 PM
Subject:Producer Properties


I started porting our code from Kafka 0.8.2.1 to 0.10.0.0 and found my 
producer code blowing up because of some changes to the config.  For 
example,  metadata.broker.list is now bootstrap.servers.  I discovered the 


ProducerConfig class which has, at least, some of the config keys.  Before 


I screw this up, I'd like some confirmation of the right mappings for 
these config pararmeters in our 0.8.2.1 code:

serializer.classMaybe value.serializer = 
VALUE_SERIALIZER_CLASS_CONFIG?
key.serializer.classMaybe key.serializer = 
KEY_SERIALIZER_CLASS_CONFIG?
producer.type   Not in ProducerConfig
batch.num.messages  Not in ProducerConfig, unless maybe 
batch.size = BATCH_SIZE_CONFIG?
queue.buffering.max.ms  Not in ProducerConfig

Thanks!

Chris

Re: Producer Properties

[Chris Barlock]

Anybody?

Chris




From:   Chris Barlock/Raleigh/IBM@IBMUS
To: users@kafka.apache.org
Date:   06/24/2016 04:56 PM
Subject:Producer Properties



I started porting our code from Kafka 0.8.2.1 to 0.10.0.0 and found my 
producer code blowing up because of some changes to the config.  For 
example,  metadata.broker.list is now bootstrap.servers.  I discovered the 

ProducerConfig class which has, at least, some of the config keys.  Before 

I screw this up, I'd like some confirmation of the right mappings for 
these config pararmeters in our 0.8.2.1 code:

serializer.classMaybe value.serializer = 
VALUE_SERIALIZER_CLASS_CONFIG?
key.serializer.classMaybe key.serializer = 
KEY_SERIALIZER_CLASS_CONFIG?
producer.type   Not in ProducerConfig
batch.num.messages  Not in ProducerConfig, unless maybe 
batch.size = BATCH_SIZE_CONFIG?
queue.buffering.max.ms  Not in ProducerConfig

Thanks!

Chris

Producer Properties

[Chris Barlock]

I started porting our code from Kafka 0.8.2.1 to 0.10.0.0 and found my 
producer code blowing up because of some changes to the config.  For 
example,  metadata.broker.list is now bootstrap.servers.  I discovered the 
ProducerConfig class which has, at least, some of the config keys.  Before 
I screw this up, I'd like some confirmation of the right mappings for 
these config pararmeters in our 0.8.2.1 code:

serializer.classMaybe value.serializer = 
VALUE_SERIALIZER_CLASS_CONFIG?
key.serializer.classMaybe key.serializer = 
KEY_SERIALIZER_CLASS_CONFIG?
producer.type   Not in ProducerConfig
batch.num.messages  Not in ProducerConfig, unless maybe 
batch.size = BATCH_SIZE_CONFIG?
queue.buffering.max.ms  Not in ProducerConfig

Thanks!

Chris

Re: AdminUtils

[Chris Barlock]

Ismael:

What is KIP-4?  Until this is available, I think I'm stuck with 
AdminUtils.  Is there any (java)doc available for it, or am I going to 
have to dig through the scala files and figure out what has changed?  The 
new REST interface is a possibility, but, if I recall, it does not 
support, at least, createTopic. 

Chris

IBM Middleware
Research Triangle Park, NC
(919) 224-2240
Internet:  barl...@us.ibm.com



From:   Ismael Juma <ism...@juma.me.uk>
To: users@kafka.apache.org
Date:   06/21/2016 05:47 PM
Subject:Re: AdminUtils
Sent by:isma...@gmail.com



Hi Chris,

Yes, `AdminUtils` is not public API. The plan is to introduce 
`AdminClient`
as part of KIP-4.

The Kafka protocol additions for `createTopic` and `deleteTopic` are
currently being discussed and it looks like they will be part of the next
Kafka release based on current progress.

The API of `AdminClient` hasn't been discussed yet, but it should be
possible to support the methods you mentioned. Maybe you can provide your
feedback as the KIP-4 discussions happen in kafka-dev?

Ismael

On Tue, Jun 21, 2016 at 10:39 PM, Chris Barlock <barl...@us.ibm.com> 
wrote:

> While working on upgrading from 0.8.2.1 to 0.10.0.0, I found out that
> AdminUtils has changed -- and not in a backwards-compatible manner.  I
> gather this is not a public API since I can't find any Javadoc for it. 
So,
> in 0.10.0.0 are there public replacements for:
>
> AdminUtils.fetchTopicMetadataFromZk
> AdminUtils.topicExists
> AdminUtils.createTopic
> AdminUtils.deleteTopic
>
> ?
>
> Thanks,
>
> Chris
>
>
>

AdminUtils

[Chris Barlock]

While working on upgrading from 0.8.2.1 to 0.10.0.0, I found out that 
AdminUtils has changed -- and not in a backwards-compatible manner.  I 
gather this is not a public API since I can't find any Javadoc for it. So, 
in 0.10.0.0 are there public replacements for:

AdminUtils.fetchTopicMetadataFromZk
AdminUtils.topicExists
AdminUtils.createTopic
AdminUtils.deleteTopic

?

Thanks,

Chris

Something Bad Happened

[Chris Barlock]

I started getting the following errors on all topics:

root@kafka-1-bw3ys:/opt/kafka/bin# ./kafka-console-consumer.sh --topic 
systemconfig --from-beginning --zookeeper $ZOOKEEPER_URL
[2016-06-20 23:43:27,460] WARN Fetching topic metadata with correlation id 
0 for topics [Set(systemconfig)] from broker 
[id:1,host:10.0.0.188,port:9092] failed (kafka.client.ClientUtils$)
java.io.EOFException: Received -1 when reading from channel, socket has 
likely been closed.
at kafka.utils.Utils$.read(Utils.scala:381)
at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at 
kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at 
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at 
kafka.network.BlockingChannel.receive(BlockingChannel.scala:111)
at 
kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at 
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at 
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2016-06-20 23:43:27,464] WARN 
[console-consumer-58602_kafka-1-bw3ys-1466466206807-f78e50cd-leader-finder-thread],
 
Failed to find leader for Set([systemconfig,1], [systemconfig,3], 
[systemconfig,0], [systemconfig,2], [systemconfig,4]) 
(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics 
[Set(systemconfig)] from broker 
[ArrayBuffer(id:1,host:10.0.0.188,port:9092)] failed
at 
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
at 
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.io.EOFException: Received -1 when reading from channel, 
socket has likely been closed.
at kafka.utils.Utils$.read(Utils.scala:381)
at 
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at 
kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at 
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at 
kafka.network.BlockingChannel.receive(BlockingChannel.scala:111)
at 
kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at 
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
... 3 more

This is with Kafka 0.8.2.1, but it also happens with 0.10.0.0.  Kafka and 
ZooKeeper are running in separate Docker containers.  I'm leaning toward a 
problem with my environment, but would love suggestions as to what might 
be causing this.

Thanks!

Chris

Re: Consumer Question

[Chris Barlock]

Thank you, but that is not the question I asked.  If a topic exists and a 
consumer is listening -- and then the topic is deleted -- does the 
consumer get some sort of notification that the topic is now gone, perhaps 
an exception?

Chris




From:   Anirudh P <panirudh2...@gmail.com>
To: users@kafka.apache.org
Date:   06/19/2016 11:04 PM
Subject:Re: Consumer Question



Hi Chris,

We should also ensure that auto.create.topics.enable is set to true.

Thank you,
Anirudh
Hi Chris,

If the topic not exist, it will create a new topic with the name which you
give.

Thanks,
Nicole

On Sat, Jun 18, 2016 at 1:55 AM, Chris Barlock <barl...@us.ibm.com> wrote:

> If you have a consumer listening on a topic and that topic is deleted is
> the consumer made aware -- perhaps by some exception -- or does it
> continue listening, blissfully unaware that it will never hear anything
> more?
>
> Thanks,
>
> Chris
>
>
>

Consumer Question

[Chris Barlock]

If you have a consumer listening on a topic and that topic is deleted is 
the consumer made aware -- perhaps by some exception -- or does it 
continue listening, blissfully unaware that it will never hear anything 
more?

Thanks, 

Chris

Re: Invalid Version for API key

[Chris Barlock]

Will have to get back to you on that, Ismael.  This code is owned by 
another team and is in a different code repository.

Chris




From:   Ismael Juma <isma...@gmail.com>
To: users@kafka.apache.org
Date:   06/08/2016 03:22 PM
Subject:Re: Invalid Version for API key



Great Chris. Out of curiosity, which Kafka client was being used by this
microservice?

Ismael
On 8 Jun 2016 18:54, "Chris Barlock" <barl...@us.ibm.com> wrote:

> Thanks, Ismael.  We tracked it down to one of the 11 microservices 
running
> on this system.
>
> Thank you, all, who replied to me!
>
> Chris
>
>
>
>
> From:   Ismael Juma <ism...@juma.me.uk>
> To: users@kafka.apache.org
> Date:   06/08/2016 11:29 AM
> Subject:Re: Invalid Version for API key
> Sent by:isma...@gmail.com
>
>
>
> Hi Chris,
>
> The error includes the IP address of the client:
>
> 172.20.3.0:53901
>
> Would that help identify the client? We are not aware of issues with the
> built-in clients.
>
> Ismael
>
> On Wed, Jun 8, 2016 at 4:13 PM, Chris Barlock <barl...@us.ibm.com> 
wrote:
>
> > I think everyone is using the Maven kafka-clients 0.8.2.1 but I can't 
be
> > sure since my team does not have control over all the code.  I cannot
> > identify what is triggering these exceptions -- which is why I was
> asking
> > about additional tracing that might add some content around the
> exception
> > to help track this down.
> >
> > Are you aware of any problems with kafka-clients 0.8.2.1 in this area?
> I'm
> > wondering if the problem could be there and that it shows up now 
because
> > of the improved error checking you mention.  In this case, Rad's
> > suggestion of upgrading the client code could resolve the issue.
> >
> > Thanks!
> >
> > Chris
> >
> >
> >
> >
> >
> > From:   Grant Henke <ghe...@cloudera.com>
> > To: users@kafka.apache.org
> > Date:   06/08/2016 10:58 AM
> > Subject:Re: Invalid Version for API key
> >
> >
> >
> > ApiKey 2 is the Offsets request. There is only a version 0 of that
> > protocol
> > since there has been no change in the protocol version for 
LIST_OFFSETS
> > from 0.8 to 0.10. So there error that version 1 is invalid is correct.
> >
> > What has changed in 0.10 is that the validation and errors of 
incorrect
> > api
> > keys has been improved.
> >
> > Are you using any custom clients? Can you identify what commands (or
> > client
> > code) trigger this error log?
> >
> >
> >
> > On Wed, Jun 8, 2016 at 9:39 AM, Chris Barlock <barl...@us.ibm.com>
> wrote:
> >
> > > Anyone?  Is there additional tracing that can be turned on to track
> down
> > > the source of these exceptions?
> > >
> > > Chris
> > >
> > >
> > >
> > >
> > > From:   Chris Barlock/Raleigh/IBM@IBMUS
> > > To: users@kafka.apache.org
> > > Date:   06/07/2016 12:45 PM
> > > Subject:Invalid Version for API key
> > >
> > >
> > >
> > > We are running some tests on upgrading from Kafka 2.10-0.8.2.1 to
> > > 2.11-0.10.00.  So far, we only upgraded the server.  All of the
> clients
> > > that I could verify are still using the Maven kafka-clients version
> > > 0.8.2.1.  There are a number of exceptions in the server.log:
> > >
> > > [2016-06-07 16:00:00,266] ERROR Closing socket for
> > > 172.20.8.19:9092-172.20.3.0:53901 because of error
> > > (kafka.network.Processor)
> > > kafka.network.InvalidRequestException: Error getting request for
> apiKey:
> > 2
> > >
> > > and apiVersion: 1
> > > at
> > >
> >
> 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:95)
> > > at
> > > kafka.network.RequestChannel$Request.(RequestChannel.scala:87)
> > > at
> > >
> > >
> >
> >
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:488)
> > > at
> > >
> > >
> >
> >
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:483)
> > > at 
scala.collection.Iterator$class.foreach(Iterator.scala:893)
> > > at
> > scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at
> sca

Re: Invalid Version for API key

[Chris Barlock]

Thanks, Ismael.  We tracked it down to one of the 11 microservices running 
on this system. 

Thank you, all, who replied to me!

Chris




From:   Ismael Juma <ism...@juma.me.uk>
To: users@kafka.apache.org
Date:   06/08/2016 11:29 AM
Subject:Re: Invalid Version for API key
Sent by:isma...@gmail.com



Hi Chris,

The error includes the IP address of the client:

172.20.3.0:53901

Would that help identify the client? We are not aware of issues with the
built-in clients.

Ismael

On Wed, Jun 8, 2016 at 4:13 PM, Chris Barlock <barl...@us.ibm.com> wrote:

> I think everyone is using the Maven kafka-clients 0.8.2.1 but I can't be
> sure since my team does not have control over all the code.  I cannot
> identify what is triggering these exceptions -- which is why I was 
asking
> about additional tracing that might add some content around the 
exception
> to help track this down.
>
> Are you aware of any problems with kafka-clients 0.8.2.1 in this area? 
I'm
> wondering if the problem could be there and that it shows up now because
> of the improved error checking you mention.  In this case, Rad's
> suggestion of upgrading the client code could resolve the issue.
>
> Thanks!
>
> Chris
>
>
>
>
>
> From:   Grant Henke <ghe...@cloudera.com>
> To: users@kafka.apache.org
> Date:   06/08/2016 10:58 AM
> Subject:Re: Invalid Version for API key
>
>
>
> ApiKey 2 is the Offsets request. There is only a version 0 of that
> protocol
> since there has been no change in the protocol version for LIST_OFFSETS
> from 0.8 to 0.10. So there error that version 1 is invalid is correct.
>
> What has changed in 0.10 is that the validation and errors of incorrect
> api
> keys has been improved.
>
> Are you using any custom clients? Can you identify what commands (or
> client
> code) trigger this error log?
>
>
>
> On Wed, Jun 8, 2016 at 9:39 AM, Chris Barlock <barl...@us.ibm.com> 
wrote:
>
> > Anyone?  Is there additional tracing that can be turned on to track 
down
> > the source of these exceptions?
> >
> > Chris
> >
> >
> >
> >
> > From:   Chris Barlock/Raleigh/IBM@IBMUS
> > To: users@kafka.apache.org
> > Date:   06/07/2016 12:45 PM
> > Subject:Invalid Version for API key
> >
> >
> >
> > We are running some tests on upgrading from Kafka 2.10-0.8.2.1 to
> > 2.11-0.10.00.  So far, we only upgraded the server.  All of the 
clients
> > that I could verify are still using the Maven kafka-clients version
> > 0.8.2.1.  There are a number of exceptions in the server.log:
> >
> > [2016-06-07 16:00:00,266] ERROR Closing socket for
> > 172.20.8.19:9092-172.20.3.0:53901 because of error
> > (kafka.network.Processor)
> > kafka.network.InvalidRequestException: Error getting request for 
apiKey:
> 2
> >
> > and apiVersion: 1
> > at
> >
> 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:95)
> > at
> > kafka.network.RequestChannel$Request.(RequestChannel.scala:87)
> > at
> >
> >
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:488)
> > at
> >
> >
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:483)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at 
scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at
> > 
kafka.network.Processor.processCompletedReceives(SocketServer.scala:483)
> > at kafka.network.Processor.run(SocketServer.scala:413)
> > at java.lang.Thread.run(Thread.java:785)
> > Caused by: java.lang.IllegalArgumentException: Invalid version for API
> key
> >
> > 2: 1
> > at
> >
> 
org.apache.kafka.common.protocol.ProtoUtils.schemaFor(ProtoUtils.java:31)
> > at
> >
> >
>
> 
org.apache.kafka.common.protocol.ProtoUtils.requestSchema(ProtoUtils.java:44)
> > at
> >
> >
>
> 
org.apache.kafka.common.protocol.ProtoUtils.parseRequest(ProtoUtils.java:60)
> > at
> >
> >
>
> 
org.apache.kafka.common.requests.ListOffsetRequest.parse(ListOffsetRequest.java:142)
> > at
> >
> >
>
> 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:46)
> > at
> >
> 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:92)
> > ... 10 more
> >
> > Any guidance on what the source of this might be?
> >
> > Chris
> >
> >
> >
> >
> >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
>
>
>
>

Re: Invalid Version for API key

[Chris Barlock]

I think everyone is using the Maven kafka-clients 0.8.2.1 but I can't be 
sure since my team does not have control over all the code.  I cannot 
identify what is triggering these exceptions -- which is why I was asking 
about additional tracing that might add some content around the exception 
to help track this down. 

Are you aware of any problems with kafka-clients 0.8.2.1 in this area? I'm 
wondering if the problem could be there and that it shows up now because 
of the improved error checking you mention.  In this case, Rad's 
suggestion of upgrading the client code could resolve the issue.

Thanks!

Chris





From:   Grant Henke <ghe...@cloudera.com>
To: users@kafka.apache.org
Date:   06/08/2016 10:58 AM
Subject:Re: Invalid Version for API key



ApiKey 2 is the Offsets request. There is only a version 0 of that 
protocol
since there has been no change in the protocol version for LIST_OFFSETS
from 0.8 to 0.10. So there error that version 1 is invalid is correct.

What has changed in 0.10 is that the validation and errors of incorrect 
api
keys has been improved.

Are you using any custom clients? Can you identify what commands (or 
client
code) trigger this error log?



On Wed, Jun 8, 2016 at 9:39 AM, Chris Barlock <barl...@us.ibm.com> wrote:

> Anyone?  Is there additional tracing that can be turned on to track down
> the source of these exceptions?
>
> Chris
>
>
>
>
> From:   Chris Barlock/Raleigh/IBM@IBMUS
> To: users@kafka.apache.org
> Date:   06/07/2016 12:45 PM
> Subject:Invalid Version for API key
>
>
>
> We are running some tests on upgrading from Kafka 2.10-0.8.2.1 to
> 2.11-0.10.00.  So far, we only upgraded the server.  All of the clients
> that I could verify are still using the Maven kafka-clients version
> 0.8.2.1.  There are a number of exceptions in the server.log:
>
> [2016-06-07 16:00:00,266] ERROR Closing socket for
> 172.20.8.19:9092-172.20.3.0:53901 because of error
> (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 
2
>
> and apiVersion: 1
> at
> 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:95)
> at
> kafka.network.RequestChannel$Request.(RequestChannel.scala:87)
> at
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:488)
> at
>
> 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:483)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.network.Processor.processCompletedReceives(SocketServer.scala:483)
> at kafka.network.Processor.run(SocketServer.scala:413)
> at java.lang.Thread.run(Thread.java:785)
> Caused by: java.lang.IllegalArgumentException: Invalid version for API 
key
>
> 2: 1
> at
> 
org.apache.kafka.common.protocol.ProtoUtils.schemaFor(ProtoUtils.java:31)
> at
>
> 
org.apache.kafka.common.protocol.ProtoUtils.requestSchema(ProtoUtils.java:44)
> at
>
> 
org.apache.kafka.common.protocol.ProtoUtils.parseRequest(ProtoUtils.java:60)
> at
>
> 
org.apache.kafka.common.requests.ListOffsetRequest.parse(ListOffsetRequest.java:142)
> at
>
> 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:46)
> at
> 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:92)
> ... 10 more
>
> Any guidance on what the source of this might be?
>
> Chris
>
>
>
>
>
>


-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: Invalid Version for API key

[Chris Barlock]

Anyone?  Is there additional tracing that can be turned on to track down 
the source of these exceptions?

Chris




From:   Chris Barlock/Raleigh/IBM@IBMUS
To: users@kafka.apache.org
Date:   06/07/2016 12:45 PM
Subject:Invalid Version for API key



We are running some tests on upgrading from Kafka 2.10-0.8.2.1 to 
2.11-0.10.00.  So far, we only upgraded the server.  All of the clients 
that I could verify are still using the Maven kafka-clients version 
0.8.2.1.  There are a number of exceptions in the server.log:

[2016-06-07 16:00:00,266] ERROR Closing socket for 
172.20.8.19:9092-172.20.3.0:53901 because of error 
(kafka.network.Processor)
kafka.network.InvalidRequestException: Error getting request for apiKey: 2 

and apiVersion: 1
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:95)
at 
kafka.network.RequestChannel$Request.(RequestChannel.scala:87)
at 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:488)
at 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:483)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at 
kafka.network.Processor.processCompletedReceives(SocketServer.scala:483)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.lang.IllegalArgumentException: Invalid version for API key 

2: 1
at 
org.apache.kafka.common.protocol.ProtoUtils.schemaFor(ProtoUtils.java:31)
at 
org.apache.kafka.common.protocol.ProtoUtils.requestSchema(ProtoUtils.java:44)
at 
org.apache.kafka.common.protocol.ProtoUtils.parseRequest(ProtoUtils.java:60)
at 
org.apache.kafka.common.requests.ListOffsetRequest.parse(ListOffsetRequest.java:142)
at 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:46)
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:92)
... 10 more

Any guidance on what the source of this might be?

Chris

Invalid Version for API key

[Chris Barlock]

We are running some tests on upgrading from Kafka 2.10-0.8.2.1 to 
2.11-0.10.00.  So far, we only upgraded the server.  All of the clients 
that I could verify are still using the Maven kafka-clients version 
0.8.2.1.  There are a number of exceptions in the server.log:

[2016-06-07 16:00:00,266] ERROR Closing socket for 
172.20.8.19:9092-172.20.3.0:53901 because of error 
(kafka.network.Processor)
kafka.network.InvalidRequestException: Error getting request for apiKey: 2 
and apiVersion: 1
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:95)
at 
kafka.network.RequestChannel$Request.(RequestChannel.scala:87)
at 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:488)
at 
kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:483)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at 
kafka.network.Processor.processCompletedReceives(SocketServer.scala:483)
at kafka.network.Processor.run(SocketServer.scala:413)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.lang.IllegalArgumentException: Invalid version for API key 
2: 1
at 
org.apache.kafka.common.protocol.ProtoUtils.schemaFor(ProtoUtils.java:31)
at 
org.apache.kafka.common.protocol.ProtoUtils.requestSchema(ProtoUtils.java:44)
at 
org.apache.kafka.common.protocol.ProtoUtils.parseRequest(ProtoUtils.java:60)
at 
org.apache.kafka.common.requests.ListOffsetRequest.parse(ListOffsetRequest.java:142)
at 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:46)
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:92)
... 10 more

Any guidance on what the source of this might be?

Chris

Re: Connection to zk shell on Kafka

[Chris Barlock]

You need the jline JAR file that ships with ZooKeeper.

Chris

IBM Tivoli Systems
Research Triangle Park, NC
(919) 224-2240
Internet:  barl...@us.ibm.com



From:   Prabhjot Bharaj prabhbha...@gmail.com
To: users@kafka.apache.org, u...@zookeeper.apache.org
Date:   07/29/2015 01:13 PM
Subject:Connection to zk shell on Kafka



Hi folks,

*/kafka/bin# ./zookeeper-shell.sh localhost:2182/*

*Connecting to localhost:2182/*

*Welcome to ZooKeeper!*

*JLine support is disabled*


 *WATCHER::*


 *WatchedEvent state:SyncConnected type:None path:null*


*The shell never says connected*

I'm running 5 node zookeeper cluster on 5-node kafka cluster (each kafka
broker has 1 zookeeper server running)

When I try connecting to the shell, the shell never says 'Connected'



However, if I try connecting on another standalone zookeeper  which has no
links to kafka, I'm able to connect:-


*/kafka/bin# /zookeeper/scripts/zkCli.sh -server 127.0.0.1:2181
http://127.0.0.1:2181*

*Connecting to 127.0.0.1:2181 http://127.0.0.1:2181*

*Welcome to ZooKeeper!*

*JLine support is enabled*


*WATCHER::*


*WatchedEvent state:SyncConnected type:None path:null*

*[zk: 127.0.0.1:2181(CONNECTED) 0]*

Am I missing something?


Thanks,

prabcs

Re: Connection to zk shell on Kafka

[Chris Barlock]

I'm a user of Kafka/ZooKeeper not one of its developers, so I can't give 
you a technical explanation.  I do agree that Kafka should ship the jline 
JAR if its zookeeper-shell depends on it.

Chris




From:   Prabhjot Bharaj prabhbha...@gmail.com
To: u...@zookeeper.apache.org, d...@kafka.apache.org
Cc: users@kafka.apache.org
Date:   07/29/2015 01:27 PM
Subject:Re: Connection to zk shell on Kafka



Sure. It would be great if you could as well explain the reason why the
absence of the jar creates this problem

Also, I'm surprised that zookeeper that comes bundled with kafka 0.8.2 
does
not have the jline jar

Regards,
prabcs

On Wed, Jul 29, 2015 at 10:45 PM, Chris Barlock barl...@us.ibm.com 
wrote:

 You need the jline JAR file that ships with ZooKeeper.

 Chris

 IBM Tivoli Systems
 Research Triangle Park, NC
 (919) 224-2240
 Internet:  barl...@us.ibm.com



 From:   Prabhjot Bharaj prabhbha...@gmail.com
 To: users@kafka.apache.org, u...@zookeeper.apache.org
 Date:   07/29/2015 01:13 PM
 Subject:Connection to zk shell on Kafka



 Hi folks,

 */kafka/bin# ./zookeeper-shell.sh localhost:2182/*

 *Connecting to localhost:2182/*

 *Welcome to ZooKeeper!*

 *JLine support is disabled*


  *WATCHER::*


  *WatchedEvent state:SyncConnected type:None path:null*


 *The shell never says connected*

 I'm running 5 node zookeeper cluster on 5-node kafka cluster (each kafka
 broker has 1 zookeeper server running)

 When I try connecting to the shell, the shell never says 'Connected'



 However, if I try connecting on another standalone zookeeper  which has 
no
 links to kafka, I'm able to connect:-


 */kafka/bin# /zookeeper/scripts/zkCli.sh -server 127.0.0.1:2181
 http://127.0.0.1:2181*

 *Connecting to 127.0.0.1:2181 http://127.0.0.1:2181*

 *Welcome to ZooKeeper!*

 *JLine support is enabled*


 *WATCHER::*


 *WatchedEvent state:SyncConnected type:None path:null*

 *[zk: 127.0.0.1:2181(CONNECTED) 0]*

 Am I missing something?


 Thanks,

 prabcs




-- 
-
There are only 10 types of people in the world: Those who understand
binary, and those who don't