Re: Kafka 8.2.2 doesn't want to compress in snappy

[Jérôme BAROTIN]

Hi,

I was 100% sure that Kafka broker didn't compress data and I didn't
think that I had to upgrade my broker to 8.2.2.

I tried the upgrade and It works right now!

I still don't understand, why the broker need to compress data again
(if the data compression is already done in the producer). Have you a
link for wiki, documentation or other to share about that?

Anyway, thanks for your help to solve this mistake.

Regards,

Jérôme

2015-10-20 1:26 GMT+02:00 Jun Rao :
> You will need to upgrade the broker to 0.8.2.2. Broker currently
> recompresses messages. In 0.8.2.1, the snappy jar has a bug that causes
> data explosion. We fixed the snappy jar in 0.8.2.2. If you upgrade the
> broker to 0.8.2.2, it will pick up the fixed snappy jar.
>
> Thanks,
>
> Jun
>
> On Sat, Oct 17, 2015 at 1:21 AM, Jérôme BAROTIN  wrote:
>
>> Hello,
>>
>> I want to check if the snappy compression works well with the java Kafka
>> client.
>>
>> In order to handle this, I set up a small program. This program
>> generate 1024 messages of readable data. Their size are of 1024 bytes
>> each. I send these messages on tree new topics and after I check the
>> size of these topic directly on the broker filesystem.
>>
>> You can find this program through the following java code :
>>
>> package unit_test.testCompress;
>>
>> import java.util.HashMap;
>> import java.util.Map;
>> import java.util.Random;
>> import java.util.concurrent.Future;
>>
>> import org.apache.kafka.clients.producer.KafkaProducer;
>> import org.apache.kafka.clients.producer.ProducerRecord;
>> import org.apache.kafka.clients.producer.RecordMetadata;
>>
>>
>> /**
>>  * Can be use in order to execute some unit test on compression
>>  */
>> public class TestCompress {
>>
>> public static void compress(String type, String version){
>> Map configs = new HashMap();
>> configs.put("key.serializer",
>> "org.apache.kafka.common.serialization.StringSerializer");
>> configs.put("producer.type", "async");
>> configs.put("compression.type", type);
>> configs.put("value.serializer",
>> "org.apache.kafka.common.serialization.ByteArraySerializer");
>> configs.put("partitioner.class",
>> "com.astellia.astkafkaproducer.RecordPartitioner");
>> configs.put("bootstrap.servers", "kafka:9092");
>>
>>
>> KafkaProducer producer = new
>> KafkaProducer(configs);
>>
>> Random r = new Random(15415485);
>> int size = 1024; //1 Ko
>> byte[] buffer = new byte[size];
>> for(int i = 0; i < size; i++){
>> buffer[i] = (byte) ('A' + (r.nextInt() % 26));
>> }
>> buffer[size-1] = 0;
>> //System.out.println(new String(buffer));
>> for(int i = 0; i < size; i++ ){
>> Future result = producer.send( new
>> ProducerRecord("unit_test_compress_"+version+ "_" +
>> type , buffer));
>> }
>>
>> producer.close();
>> }
>>
>> public static void main(String[] args) {
>>
>> String version = "v10";
>> compress("snappy",version);
>> compress("gzip",version);
>> compress("none",version);
>>
>> }
>>
>> }
>>
>>
>> I'm compiling this code with this following maven pom file :
>>
>> http://maven.apache.org/POM/4.0.0;
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>>   4.0.0
>>
>>   unit_test
>>   testCompress
>>   0.0.1-SNAPSHOT
>>   jar
>>
>>   testCompress
>>   http://maven.apache.org
>>
>>   
>> UTF-8
>>   
>>
>>   
>>  
>> org.apache.kafka
>> kafka_2.10
>> 0.8.2.2
>> 
>>   
>> 
>>
>> This program executes very well on my computer.
>>
>> But when I check the results directly on my kafka broker through the
>> command line tool "du" the space took by each topics. I found  :
>> - gzip topic is compressed that's ok
>> - none topic is not compressed that's ok
>> - but snappy topic is not compressed, that's not ok
>> (screenshot can be found here : http://i.stack.imgur.com/7W1f5.png)
>>
>>
>> I checked though vi the stored file and data are still clear.
>>
>> I'm aware about this issue on Kafka 8.2.1 :
>> https://issues.apache.org/jira/browse/KAFKA-2189
>>
>> But I'm using Kafka 8.2.2 on producer and kafka 8.2.1 on broker.
>>
>> I checked the dependency of Snappy as well. I'm using the 1.1.1.7
>>
>> Have you an idea of how to enable snappy compression on Kafak ?
>> Did I forget a parameter to enable snappy compression on kafka ?
>> Are my kafka version not compatible ?
>>

Re: Getting error while reading message

[Selina Tech]

I got same error when I send message to Kafka. generally it caused by
deserializer
as Hemant mentioned.
You need to check how the data was send to Kafka, and how is your consumer
deserializer defined.
And you need to check for both Key and value.

the data is the topic might be byte[] type

Sincerely,
Selina

On Sun, Oct 18, 2015 at 3:01 AM, Kiran Singh  wrote:

> Hi
>
> I am trying to get message from kafka server using hig level consumer API.
> But i am getting following exception
>
> "java.lang.ClassCastException: [B cannot be cast to java.lang.String"
>
> Can anyone explain what this means.
>
> Thanks
> Kiran Singh
>

Kafka - Rest api query

[Kudumula, Surender]

Dear sir/madam
I have a query. We are working on POC at the moment and we are using kafka to 
produce and consume messages. I have one component which consumes the request 
from topic and processes it and creates a file and again produce the java 
object as byte array to another kafka topic. Now I have two options

1. Is to write a java rest client and send the bytearray java object to 
rest api service in another component

2. Or I was wondering if its possible to route the message from kafka topic 
to another component rest APi server.

Please let me know we are using kafka 0.8 and new producer. Any suggestions 
would be appreciated?

Regards

Surender Kudumula
Big Data Consultant - EMEA
Analytics & Data Management

surender.kudum...@hpe.com
M +44 7795970923

Hewlett-Packard Enterprise
Cain Rd,
Bracknell
RG12 1HN
UK

[http://graphics8.nytimes.com/images/2015/06/03/technology/03bits-hp/03bits-hp-master315.png]

Very high count

[Tobias Heintz]

For our Kafka cluster of three machines, we have set up a Grafana dashboard, 
which, using JMX through collectd, holds some under the hood metrics of the 
cluster. Especially interesting is the "kafka-messages-in-per-sec" metric, 
which appears to be the number of messages the cluster takes in per second. 
However, this metric is way too high for our use case, reporting almost 10x the 
number of messages per sec that our old proprietary messaging platform reports.

My question: what exactly does this number represent? Which kinds of messages 
are measured here? How can such a great gap be explained (we have very high 
trust in our old platform that it is reporting the proper values)?

Thanks a lot
tobias

--
Tobias Heintz
Teamlead Core

Telefon: +49 30 47375370 | Fax: +49 30 484984411
E-Mail: tobias.hei...@plista.com
Links: www.plista.com

plista GmbH | Torstraße 33-35 | 10119 Berlin | Deutschland
Amtsgericht Berlin-Charlottenburg | HRB 114726 B
Geschäftsführer: Andreas Richter | Christian Laase | Dr. Dominik Matyka | Jana 
Kusick

Re: Kafka - Rest api query

[Sharninder]

Sounds like an app design decision. What help can this list give you ? 


> On 20-Oct-2015, at 8:07 PM, Kudumula, Surender  
> wrote:
> 
> Dear sir/madam
> I have a query. We are working on POC at the moment and we are using kafka to 
> produce and consume messages. I have one component which consumes the request 
> from topic and processes it and creates a file and again produce the java 
> object as byte array to another kafka topic. Now I have two options
> 1. Is to write a java rest client and send the bytearray java object to 
> rest api service in another component
> 2. Or I was wondering if its possible to route the message from kafka 
> topic to another component rest APi server.
>  
> Please let me know we are using kafka 0.8 and new producer. Any suggestions 
> would be appreciated?
>  
> Regards
>  
> Surender Kudumula
> Big Data Consultant - EMEA
> Analytics & Data Management
>  
> surender.kudum...@hpe.com
> M +44 7795970923
> 
> Hewlett-Packard Enterprise
> Cain Rd,
> Bracknell
> RG12 1HN
> UK
> 
> 
>  

Re: Kafka topic message consumer fetch response time checking

[Ewen Cheslack-Postava]

If you want the full round-trip latency, you need to measure that at the
client. The performance measurement tools should do this pretty accurately.
For example, if you just want to know how long it takes to produce a
message to Kafka and get an ack back, you can use the latency numbers
reported by ProducerPerformance, ensuring to use a low throughput to avoid
queuing (so you measure the latency rather than queuing delay). If you
wanted producer -> brokers -> consumer latency, use EndToEndLatency.

There is some startup cost, but they don't start timing anything until a
lot of the startup has already taken place and as long as you run a long
enough test, the impact of the initial startup cost shouldn't be noticeable.

-Ewen

On Mon, Oct 19, 2015 at 2:45 PM, David Luu  wrote:

> If one wanted to check response time for a consumer fetching a topic
> message (on the client side), similar to checking an HTTP request's
> response time for the web, what's the best approach to take?
>
> I notice the kafka shell scripts if used for that have some startup
> overhead if used to assess response times frequently.
>
> Is there a speedier CLI alternative, or will one have to write a custom
> script/program in one of the kafka language clients for less startup
> overhead for frequent periodic checks?
>
> Or is there a kafka server side metric that is sufficient to check for this
> (no need for client side checking, just look at the server metrics)?
>



-- 
Thanks,
Ewen

Kafka 0.8.2.1 OffsetCommitRequests and auto.commit=true

[William Grim]

Hello, Kafka users!

I've been trying to build Apache Kafka into our system, and for one part of
our system, we have a use-case where we have to leave auto-commit enabled
but would like to reset the offset numbers to an earlier offset if a
failure happens in our code.  We are using auto-commit because the 0.8.x
branch (0.8.2.1 for us, atm) has a bug in the commitOffsets logic and
doesn't retry like it should; although, this has been fixed in the
unreleased master branch.

Anyway, this is our issue:

   1. Streaming data from kafka using a high level consumer
   2. Failure happens on our end:
  1. Initiate shutdown of the high level consumer/reader.
  2. Send a OffsetCommitRequest using a SimpleConsumer connected to the
  offset coordinator.for the very same consumer group that the high level
  consumer/reader was using.
   3. Offsets are then reset.
   4. A few seconds later, offsets get changed back to the latest
   automatically.

I suspect #4 happens due to autocommit logic, but why?  As far as I can
tell, we shut down our consumer that had known about the higher offsets.
Plus, I thought once I sent an OffsetCommitRequest for a given consumer
group, that information should be flowed to the rest of the consumers in
the consumer group.

Is this another bug, or am I misunderstanding something about Kafka?

-- 




*William GrimSr. Software Engineerm: 914 418 4115
<914%20418%204115>e: wg...@signal.co signal.co
Cut
Through the NoiseThis e-mail and any files transmitted with it are for the
sole use of the intended recipient(s) and may contain confidential and
privileged information. Any unauthorized use of this email is strictly
prohibited. ©2015 Signal. All rights reserved.*

New Kafka consumer stability

[tao xiao]

Hi,

I am starting a new project that requires heavy use on Kafka consumer. I
did a quick look at the new Kafka consumer and found it provides some of
the features we definitely need. But as it is annotated as unable is it
safe to rely on it or it will still be evolving dramatically in coming
releases?

Re: kafka consumer shell scripts (and kafkacat) with respect to JSON pretty printing

[Ewen Cheslack-Postava]

You can accomplish this with the console consumer -- it has a formatter
flag that lets you plug in custom logic for formatting messages. The
default does not do any formatting, but if you write your own
implementation, you just need to set the flag to plug it in.

You can see an example of this in Confluent's Avro support:
https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/formatter/AvroMessageFormatter.java
This deserializes using Avro and writes using Avro's JSON encoding to get
nice, human-readable output. There are also wrapper scripts included to set
the appropriate flags so it is as easy to use as the normal console
consumer scripts.

If you'd like something similar, our next release will have JSON
serializers (see
https://github.com/confluentinc/schema-registry/tree/master/json-serializer)
but does not yet include the corresponding formatter. However, the
formatter implementation should look very similar to the deserializer.

-Ewen

-Ewen


On Fri, Oct 16, 2015 at 7:10 PM, David Luu  wrote:

> I was wondering, do the kafka consumer shell scripts (high and low level
> ones) and kafkacat do any pre-processing of the topic messages before
> outputting to stdout or does it just output "as is" in the format the
> message originally came in through kafka from the producer?
>
> Meaning pretty printed JSON produced is consumed as pretty printed JSON,
> line delimited JSON blobs (not pretty printed) is consumed the same way.
>
> I'm asking this as I notice some topics I'm consuming with the shell
> scripts (and kafkacat) are pretty printed and some not. So just wanted to
> confirm this while also checking with the developers of the topic producers
> on whether they are pretty printing on their end or not.
>
> In general, I'm assuming as best practice, it's better not to pretty print
> the JSON as a producer to save on message size for network transmission and
> file storage by kafka, since those extra newlines and spaces/tabs add up
> over time?
>



-- 
Thanks,
Ewen

Re: Avro messages from Kafka topic

[Rakesh Vidyadharan]

We publish messages to kafka in Thrift format.  We use the old simple
consumer and just retrieve the message bytes, transform back to object
model using Thrift API and do whatever our application needs with it.

On 20/10/2015 11:08, "Buntu Dev"  wrote:

>I got a Kafka topic with messages in Avro format. I would like to display
>the live stream of these events in a web app. Are there any streaming
>consumer clients to convert the Avro messages into some readable format,
>if
>not any insight into how I can achieve this would be very helpful?
>
>Thanks!

Avro messages from Kafka topic

[Buntu Dev]

I got a Kafka topic with messages in Avro format. I would like to display
the live stream of these events in a web app. Are there any streaming
consumer clients to convert the Avro messages into some readable format, if
not any insight into how I can achieve this would be very helpful?

Thanks!

Failed to read complete buffer while using simpleApi fetch message from broker

[Gene Gao]

hi there
 we're running kafka cluster with 10 brokers and two topics, each topics 
has 500 partitions(kafka version is 0.8.2.1), when we start a hadoop job to 
fetch message from cluster(one hadoop map for one partition), 499/500 were 
successed, only one task fail. And the error on that broker is like this(I 
found it in server.log):


[2015-10-20 19:00:02,996] ERROR [Replica Manager on Broker 10]: Error when 
processing fetch request for partition [ualog,54] offset 109455388 from 
consumer with correlation id 0. Possible cause: Failed to read complete buffer 
for targetOffset 110982779 startPosition 2147469523 in 
/disk1/data/ualog-54/000106895524.log (kafka.server.ReplicaManager)



My code will keeping try to fetch the specificed offset message until 
hadoop task timeout , and of course the broker keep give me this error log but 
nothing changes. I have no idea what's that means.
So I hope you guys can help me out here. 
thanks

Re: It's 5.41am, we're after 20+ hours of debugging our prod cluster. See NotAssignedReplicaException and UnknownException errors. Help?

[Shaun Senecal]

I can't say this is the same issue, but it sounds similar to a situation we 
experienced with Kafka 0.8.2.[1-2].  After restarting a broker, the cluster 
would never really recover (ISRs constantly changing, replication failing, 
etc).  We found the only way to fully recover the cluster was to stop all 
producers and consumers, restart the kafka cluster, the once the cluster was 
back up, restart the producers/consumers.  Obviously thats not acceptable for a 
production cluster, but that was the only thing we could find that would get us 
going again.


Shaun


From: Szymon Sobczak 
Sent: October 19, 2015 9:52 PM
To: users@kafka.apache.org
Cc: Big Data
Subject: It's 5.41am, we're after 20+ hours of debugging our prod cluster. See 
NotAssignedReplicaException and UnknownException errors. Help?

Hi!

We're running a 5-machine production Kafka cluster on version 0.8.1.1.
Yesterday we had some disk problems on one of the replicas and decided to
replace that node with a clean one. That's when we started experiencing
many different problems:

- partition replicas are still assigned to the old node and we can't remove
it form the replica list
- replicas are lagging behind, most of the topics have only one ISR
- most of the leaders are on a single node
- CPU load on the machines is constantly high

We've tried to rebalance the cluster by moving the leaders, decreasing
number of replicas and some others, but it doesn't seem to help. In the
meantime I've noticed very weird errors in the kafka.log

For partition 0 of topic product_templates with the following description:

Topic:product_templates PartitionCount:2 ReplicationFactor:3 Configs:
Topic: product_templates Partition: 0 Leader: 135 Replicas: 135,163,68 Isr:
135,68,163
Topic: product_templates Partition: 1 Leader: 155 Replicas: 163,68,164 Isr:
155,68,164

On machine 135 (which is a leader of product_templates,0) in kafka.log I
see:

kafka.common.NotAssignedReplicaException: Leader 135 failed to record
follower 155's position 0 for partition [product_templates,0] since the
replica 155 is not recognized to be one of the assigned replicas 68,163,135
for partition [product_templates,0]

And the complimentary, on 155 - NOT a replica product_templates,0:

ERROR [ReplicaFetcherThread-0-135] 2015-10-20 04:41:47,011 Logging.scala
kafka.server.ReplicaFetcherThread [ReplicaFetcherThread-0-135], Error for
partition [product_templates,0] to broker 135:class
kafka.common.UnknownException

Both of those happen for multiple topics, on multiple machines. Every
single one happens multiple times per second...

How to approach this? Any help is appreciated!

Thanks!
Szymon.

Re: Kafka 8.2.2 doesn't want to compress in snappy

[Gwen Shapira]

We compress a batch of messages together, but we need to give each
message its own offset (and know its key if we want to use topic
compaction), so messages are un-compressed and re-compressed.

We are working on an improvement to add relative offsets which will
allow the broker to skip this re-compression.

Gwen

On Tue, Oct 20, 2015 at 3:18 AM, Jérôme BAROTIN  wrote:
> Hi,
>
> I was 100% sure that Kafka broker didn't compress data and I didn't
> think that I had to upgrade my broker to 8.2.2.
>
> I tried the upgrade and It works right now!
>
> I still don't understand, why the broker need to compress data again
> (if the data compression is already done in the producer). Have you a
> link for wiki, documentation or other to share about that?
>
> Anyway, thanks for your help to solve this mistake.
>
> Regards,
>
> Jérôme
>
> 2015-10-20 1:26 GMT+02:00 Jun Rao :
>> You will need to upgrade the broker to 0.8.2.2. Broker currently
>> recompresses messages. In 0.8.2.1, the snappy jar has a bug that causes
>> data explosion. We fixed the snappy jar in 0.8.2.2. If you upgrade the
>> broker to 0.8.2.2, it will pick up the fixed snappy jar.
>>
>> Thanks,
>>
>> Jun
>>
>> On Sat, Oct 17, 2015 at 1:21 AM, Jérôme BAROTIN  wrote:
>>
>>> Hello,
>>>
>>> I want to check if the snappy compression works well with the java Kafka
>>> client.
>>>
>>> In order to handle this, I set up a small program. This program
>>> generate 1024 messages of readable data. Their size are of 1024 bytes
>>> each. I send these messages on tree new topics and after I check the
>>> size of these topic directly on the broker filesystem.
>>>
>>> You can find this program through the following java code :
>>>
>>> package unit_test.testCompress;
>>>
>>> import java.util.HashMap;
>>> import java.util.Map;
>>> import java.util.Random;
>>> import java.util.concurrent.Future;
>>>
>>> import org.apache.kafka.clients.producer.KafkaProducer;
>>> import org.apache.kafka.clients.producer.ProducerRecord;
>>> import org.apache.kafka.clients.producer.RecordMetadata;
>>>
>>>
>>> /**
>>>  * Can be use in order to execute some unit test on compression
>>>  */
>>> public class TestCompress {
>>>
>>> public static void compress(String type, String version){
>>> Map configs = new HashMap();
>>> configs.put("key.serializer",
>>> "org.apache.kafka.common.serialization.StringSerializer");
>>> configs.put("producer.type", "async");
>>> configs.put("compression.type", type);
>>> configs.put("value.serializer",
>>> "org.apache.kafka.common.serialization.ByteArraySerializer");
>>> configs.put("partitioner.class",
>>> "com.astellia.astkafkaproducer.RecordPartitioner");
>>> configs.put("bootstrap.servers", "kafka:9092");
>>>
>>>
>>> KafkaProducer producer = new
>>> KafkaProducer(configs);
>>>
>>> Random r = new Random(15415485);
>>> int size = 1024; //1 Ko
>>> byte[] buffer = new byte[size];
>>> for(int i = 0; i < size; i++){
>>> buffer[i] = (byte) ('A' + (r.nextInt() % 26));
>>> }
>>> buffer[size-1] = 0;
>>> //System.out.println(new String(buffer));
>>> for(int i = 0; i < size; i++ ){
>>> Future result = producer.send( new
>>> ProducerRecord("unit_test_compress_"+version+ "_" +
>>> type , buffer));
>>> }
>>>
>>> producer.close();
>>> }
>>>
>>> public static void main(String[] args) {
>>>
>>> String version = "v10";
>>> compress("snappy",version);
>>> compress("gzip",version);
>>> compress("none",version);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> I'm compiling this code with this following maven pom file :
>>>
>>> http://maven.apache.org/POM/4.0.0;
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>>   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>>>   4.0.0
>>>
>>>   unit_test
>>>   testCompress
>>>   0.0.1-SNAPSHOT
>>>   jar
>>>
>>>   testCompress
>>>   http://maven.apache.org
>>>
>>>   
>>> UTF-8
>>>   
>>>
>>>   
>>>  
>>> org.apache.kafka
>>> kafka_2.10
>>> 0.8.2.2
>>> 
>>>   
>>> 
>>>
>>> This program executes very well on my computer.
>>>
>>> But when I check the results directly on my kafka broker through the
>>> command line tool "du" the space took by each topics. I found  :
>>> - gzip topic is compressed that's ok
>>> - none topic is not compressed that's ok
>>> - but snappy topic is not compressed, that's not ok
>>> (screenshot can be found here : http://i.stack.imgur.com/7W1f5.png)

Re: New Kafka consumer stability

[Guozhang Wang]

Tao,

The APIs should not be evolving dramatically after the 0.9.0 release.

Stable-wise, we are doing a bunch of system / integration tests right now
to make sure it is in a good shape upon release. But since it is the first
release one cannot guarantee it is completely bug-free.

Guozhang

On Tue, Oct 20, 2015 at 12:16 PM, tao xiao  wrote:

> Hi,
>
> I am starting a new project that requires heavy use on Kafka consumer. I
> did a quick look at the new Kafka consumer and found it provides some of
> the features we definitely need. But as it is annotated as unable is it
> safe to rely on it or it will still be evolving dramatically in coming
> releases?
>



-- 
-- Guozhang

RE: Kafka - Rest api query

[Kudumula, Surender]

Thanks for the reply. Iam looking to know if its possible to route binary 
objects messages to rest api service from kafka. If so please let me know. 
Otherwise I can consume the binary object using java consumer and then create a 
rest client and send the binary message via HTTP POST to rest server. 
Appreciate your suggestions. Thanks

Regards

Surender Kudumula
Big Data Architect - EMEA
Analytics & Data Management

surender.kudum...@hp.com
M +44 7795970923

Hewlett-Packard Company
88 Wood St
London EC2V 7QT
UK



-Original Message-
From: Sharninder [mailto:sharnin...@gmail.com] 
Sent: 20 October 2015 16:35
To: users@kafka.apache.org
Subject: Re: Kafka - Rest api query

Sounds like an app design decision. What help can this list give you ? 


> On 20-Oct-2015, at 8:07 PM, Kudumula, Surender  
> wrote:
> 
> Dear sir/madam
> I have a query. We are working on POC at the moment and we are using kafka to 
> produce and consume messages. I have one component which consumes the request 
> from topic and processes it and creates a file and again produce the java 
> object as byte array to another kafka topic. Now I have two options
> 1. Is to write a java rest client and send the bytearray java object to 
> rest api service in another component
> 2. Or I was wondering if its possible to route the message from kafka 
> topic to another component rest APi server.
>  
> Please let me know we are using kafka 0.8 and new producer. Any suggestions 
> would be appreciated?
>  
> Regards
>  
> Surender Kudumula
> Big Data Consultant - EMEA
> Analytics & Data Management
>  
> surender.kudum...@hpe.com
> M +44 7795970923
> 
> Hewlett-Packard Enterprise
> Cain Rd,
> Bracknell
> RG12 1HN
> UK
> 
> 
>  

Re: Kafka - Rest api query

[Selina Tech]

Could you write a consumer at your rest server?

On Tue, Oct 20, 2015 at 1:18 PM, Kudumula, Surender <
surender.kudum...@hpe.com> wrote:

> Thanks for the reply. Iam looking to know if its possible to route binary
> objects messages to rest api service from kafka. If so please let me know.
> Otherwise I can consume the binary object using java consumer and then
> create a rest client and send the binary message via HTTP POST to rest
> server. Appreciate your suggestions. Thanks
>
> Regards
>
> Surender Kudumula
> Big Data Architect - EMEA
> Analytics & Data Management
>
> surender.kudum...@hp.com
> M +44 7795970923
>
> Hewlett-Packard Company
> 88 Wood St
> London EC2V 7QT
> UK
>
>
>
> -Original Message-
> From: Sharninder [mailto:sharnin...@gmail.com]
> Sent: 20 October 2015 16:35
> To: users@kafka.apache.org
> Subject: Re: Kafka - Rest api query
>
> Sounds like an app design decision. What help can this list give you ?
>
>
> > On 20-Oct-2015, at 8:07 PM, Kudumula, Surender <
> surender.kudum...@hpe.com> wrote:
> >
> > Dear sir/madam
> > I have a query. We are working on POC at the moment and we are using
> kafka to produce and consume messages. I have one component which consumes
> the request from topic and processes it and creates a file and again
> produce the java object as byte array to another kafka topic. Now I have
> two options
> > 1. Is to write a java rest client and send the bytearray java object
> to rest api service in another component
> > 2. Or I was wondering if its possible to route the message from
> kafka topic to another component rest APi server.
> >
> > Please let me know we are using kafka 0.8 and new producer. Any
> suggestions would be appreciated?
> >
> > Regards
> >
> > Surender Kudumula
> > Big Data Consultant - EMEA
> > Analytics & Data Management
> >
> > surender.kudum...@hpe.com
> > M +44 7795970923
> >
> > Hewlett-Packard Enterprise
> > Cain Rd,
> > Bracknell
> > RG12 1HN
> > UK
> >
> >
> >
>

RE: Kafka - Rest api query

[Kudumula, Surender]

Actually iam planning to write a consumer in a rest client where kafka topic is 
residing and send the object from rest client to another webservice which 
accepts rest api service.

Regards

Surender Kudumula
Big Data Consultant - EMEA
Analytics & Data Management

surender.kudum...@hpe.com
M +44 7795970923

Hewlett-Packard Enterprise
Cain Rd, 
Bracknell 
RG12 1HN
UK



-Original Message-
From: Selina Tech [mailto:swucaree...@gmail.com] 
Sent: 20 October 2015 21:50
To: users@kafka.apache.org
Subject: Re: Kafka - Rest api query

Could you write a consumer at your rest server?

On Tue, Oct 20, 2015 at 1:18 PM, Kudumula, Surender < 
surender.kudum...@hpe.com> wrote:

> Thanks for the reply. Iam looking to know if its possible to route 
> binary objects messages to rest api service from kafka. If so please let me 
> know.
> Otherwise I can consume the binary object using java consumer and then 
> create a rest client and send the binary message via HTTP POST to rest 
> server. Appreciate your suggestions. Thanks
>
> Regards
>
> Surender Kudumula
> Big Data Architect - EMEA
> Analytics & Data Management
>
> surender.kudum...@hp.com
> M +44 7795970923
>
> Hewlett-Packard Company
> 88 Wood St
> London EC2V 7QT
> UK
>
>
>
> -Original Message-
> From: Sharninder [mailto:sharnin...@gmail.com]
> Sent: 20 October 2015 16:35
> To: users@kafka.apache.org
> Subject: Re: Kafka - Rest api query
>
> Sounds like an app design decision. What help can this list give you ?
>
>
> > On 20-Oct-2015, at 8:07 PM, Kudumula, Surender <
> surender.kudum...@hpe.com> wrote:
> >
> > Dear sir/madam
> > I have a query. We are working on POC at the moment and we are using
> kafka to produce and consume messages. I have one component which 
> consumes the request from topic and processes it and creates a file 
> and again produce the java object as byte array to another kafka 
> topic. Now I have two options
> > 1. Is to write a java rest client and send the bytearray java object
> to rest api service in another component
> > 2. Or I was wondering if its possible to route the message from
> kafka topic to another component rest APi server.
> >
> > Please let me know we are using kafka 0.8 and new producer. Any
> suggestions would be appreciated?
> >
> > Regards
> >
> > Surender Kudumula
> > Big Data Consultant - EMEA
> > Analytics & Data Management
> >
> > surender.kudum...@hpe.com
> > M +44 7795970923
> >
> > Hewlett-Packard Enterprise
> > Cain Rd,
> > Bracknell
> > RG12 1HN
> > UK
> >
> >
> >
>

Re: Client consumer question

[Mohit Anchlia]

Is there a wiki page where I can find all the major design changes in 0.9.0?

On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang  wrote:

> It is not released yet, we are shooting for Nov. for 0.9.0.
>
> Guozhang
>
> On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia 
> wrote:
>
> > Is 0.9.0 still under development? I don't see it here:
> > http://kafka.apache.org/downloads.html
> >
> > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang 
> wrote:
> >
> > > The links you are referring are for the old consumer.
> > >
> > > If you are using the ZooKeeper based high-level version of the old
> > consumer
> > > which is described in the second link, then failures are handled and
> > > abstracted from you so that if there is a failure in the current
> process,
> > > its fetching partitions will be re-assigned to other consumers within
> the
> > > same group starting at the last checkpointed offset. And offsets can be
> > > either checkpointed periodically or manually throw consumer.commit()
> > calls.
> > >
> > > BTW, in the coming 0.9.0 release there is a new consumer written in
> Java
> > > which uses a poll() based API instead of a stream iterating API. More
> > > details can be found here in case you are interested in trying it out:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >
> > > Guozhang
> > >
> > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia  >
> > > wrote:
> > >
> > > > By old consumer you mean version < .8?
> > > >
> > > > Here are the links:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > >
> > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang 
> > > > wrote:
> > > >
> > > > > Hi Mohit,
> > > > >
> > > > > Are you referring to the new Java consumer or the old consumer? Or
> > more
> > > > > specifically what examples doc are you referring to?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > mohitanch...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I see most of the consumer examples create a while/for loop and
> > then
> > > > > fetch
> > > > > > messages iteratively. Is that the only way by which clients can
> > > > consumer
> > > > > > messages? If this is the preferred way then how do you deal with
> > > > > failures,
> > > > > > exceptions such that messages are not lost.
> > > > > >
> > > > > > Also, please point me to examples that one would consider as a
> > robust
> > > > way
> > > > > > of coding consumers.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Client consumer question

[Guozhang Wang]

We will have a release document for that on the release date, it is not
complete yet.

Guozhang

On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia 
wrote:

> Is there a wiki page where I can find all the major design changes in
> 0.9.0?
>
> On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang  wrote:
>
> > It is not released yet, we are shooting for Nov. for 0.9.0.
> >
> > Guozhang
> >
> > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia 
> > wrote:
> >
> > > Is 0.9.0 still under development? I don't see it here:
> > > http://kafka.apache.org/downloads.html
> > >
> > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang 
> > wrote:
> > >
> > > > The links you are referring are for the old consumer.
> > > >
> > > > If you are using the ZooKeeper based high-level version of the old
> > > consumer
> > > > which is described in the second link, then failures are handled and
> > > > abstracted from you so that if there is a failure in the current
> > process,
> > > > its fetching partitions will be re-assigned to other consumers within
> > the
> > > > same group starting at the last checkpointed offset. And offsets can
> be
> > > > either checkpointed periodically or manually throw consumer.commit()
> > > calls.
> > > >
> > > > BTW, in the coming 0.9.0 release there is a new consumer written in
> > Java
> > > > which uses a poll() based API instead of a stream iterating API. More
> > > > details can be found here in case you are interested in trying it
> out:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> mohitanch...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > By old consumer you mean version < .8?
> > > > >
> > > > > Here are the links:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > >
> > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> wangg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Mohit,
> > > > > >
> > > > > > Are you referring to the new Java consumer or the old consumer?
> Or
> > > more
> > > > > > specifically what examples doc are you referring to?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > mohitanch...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I see most of the consumer examples create a while/for loop and
> > > then
> > > > > > fetch
> > > > > > > messages iteratively. Is that the only way by which clients can
> > > > > consumer
> > > > > > > messages? If this is the preferred way then how do you deal
> with
> > > > > > failures,
> > > > > > > exceptions such that messages are not lost.
> > > > > > >
> > > > > > > Also, please point me to examples that one would consider as a
> > > robust
> > > > > way
> > > > > > > of coding consumers.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: Client consumer question

[Mohit Anchlia]

Thanks. Are there any other major changes in .9 release other than the
Consumer changes. Should I wait for .9 or go ahead and performance test
with .8?

On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang  wrote:

> We will have a release document for that on the release date, it is not
> complete yet.
>
> Guozhang
>
> On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia 
> wrote:
>
> > Is there a wiki page where I can find all the major design changes in
> > 0.9.0?
> >
> > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang 
> wrote:
> >
> > > It is not released yet, we are shooting for Nov. for 0.9.0.
> > >
> > > Guozhang
> > >
> > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia  >
> > > wrote:
> > >
> > > > Is 0.9.0 still under development? I don't see it here:
> > > > http://kafka.apache.org/downloads.html
> > > >
> > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > The links you are referring are for the old consumer.
> > > > >
> > > > > If you are using the ZooKeeper based high-level version of the old
> > > > consumer
> > > > > which is described in the second link, then failures are handled
> and
> > > > > abstracted from you so that if there is a failure in the current
> > > process,
> > > > > its fetching partitions will be re-assigned to other consumers
> within
> > > the
> > > > > same group starting at the last checkpointed offset. And offsets
> can
> > be
> > > > > either checkpointed periodically or manually throw
> consumer.commit()
> > > > calls.
> > > > >
> > > > > BTW, in the coming 0.9.0 release there is a new consumer written in
> > > Java
> > > > > which uses a poll() based API instead of a stream iterating API.
> More
> > > > > details can be found here in case you are interested in trying it
> > out:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> > mohitanch...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > By old consumer you mean version < .8?
> > > > > >
> > > > > > Here are the links:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> > wangg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mohit,
> > > > > > >
> > > > > > > Are you referring to the new Java consumer or the old consumer?
> > Or
> > > > more
> > > > > > > specifically what examples doc are you referring to?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > > mohitanch...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I see most of the consumer examples create a while/for loop
> and
> > > > then
> > > > > > > fetch
> > > > > > > > messages iteratively. Is that the only way by which clients
> can
> > > > > > consumer
> > > > > > > > messages? If this is the preferred way then how do you deal
> > with
> > > > > > > failures,
> > > > > > > > exceptions such that messages are not lost.
> > > > > > > >
> > > > > > > > Also, please point me to examples that one would consider as
> a
> > > > robust
> > > > > > way
> > > > > > > > of coding consumers.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Client consumer question

[Guozhang Wang]

There are a bunch of new features added in 0.9 plus quite a lot of bug
fixes as well, a complete ticket list can be found here:

https://issues.apache.org/jira/browse/KAFKA-1686?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.9.0.0%20ORDER%20BY%20updated%20DESC

In a short summary of the new features, 0.9 introduces:

1) security and quota management on the brokers.
2) new Java consumer.
3) copycat framework for ingress / egress of Kafka.

Guozhang

On Tue, Oct 20, 2015 at 4:32 PM, Mohit Anchlia 
wrote:

> Thanks. Are there any other major changes in .9 release other than the
> Consumer changes. Should I wait for .9 or go ahead and performance test
> with .8?
>
> On Tue, Oct 20, 2015 at 3:54 PM, Guozhang Wang  wrote:
>
> > We will have a release document for that on the release date, it is not
> > complete yet.
> >
> > Guozhang
> >
> > On Tue, Oct 20, 2015 at 3:18 PM, Mohit Anchlia 
> > wrote:
> >
> > > Is there a wiki page where I can find all the major design changes in
> > > 0.9.0?
> > >
> > > On Mon, Oct 19, 2015 at 4:24 PM, Guozhang Wang 
> > wrote:
> > >
> > > > It is not released yet, we are shooting for Nov. for 0.9.0.
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia <
> mohitanch...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Is 0.9.0 still under development? I don't see it here:
> > > > > http://kafka.apache.org/downloads.html
> > > > >
> > > > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > The links you are referring are for the old consumer.
> > > > > >
> > > > > > If you are using the ZooKeeper based high-level version of the
> old
> > > > > consumer
> > > > > > which is described in the second link, then failures are handled
> > and
> > > > > > abstracted from you so that if there is a failure in the current
> > > > process,
> > > > > > its fetching partitions will be re-assigned to other consumers
> > within
> > > > the
> > > > > > same group starting at the last checkpointed offset. And offsets
> > can
> > > be
> > > > > > either checkpointed periodically or manually throw
> > consumer.commit()
> > > > > calls.
> > > > > >
> > > > > > BTW, in the coming 0.9.0 release there is a new consumer written
> in
> > > > Java
> > > > > > which uses a poll() based API instead of a stream iterating API.
> > More
> > > > > > details can be found here in case you are interested in trying it
> > > out:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Oct 19, 2015 at 2:54 PM, Mohit Anchlia <
> > > mohitanch...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > By old consumer you mean version < .8?
> > > > > > >
> > > > > > > Here are the links:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > > >
> > > > > > > On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang <
> > > wangg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mohit,
> > > > > > > >
> > > > > > > > Are you referring to the new Java consumer or the old
> consumer?
> > > Or
> > > > > more
> > > > > > > > specifically what examples doc are you referring to?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia <
> > > > > > mohitanch...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I see most of the consumer examples create a while/for loop
> > and
> > > > > then
> > > > > > > > fetch
> > > > > > > > > messages iteratively. Is that the only way by which clients
> > can
> > > > > > > consumer
> > > > > > > > > messages? If this is the preferred way then how do you deal
> > > with
> > > > > > > > failures,
> > > > > > > > > exceptions such that messages are not lost.
> > > > > > > > >
> > > > > > > > > Also, please point me to examples that one would consider
> as
> > a
> > > > > robust
> > > > > > > way
> > > > > > > > > of coding consumers.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang