I am trying to build a POC with Kafka 0.8.1. I am using my own java class as a
Kafka message which has a bunch of String data types. For serializer.class
property in my producer, I cannot use the default serializer class or the
String serializer class that comes with Kafka library. I guess I
you can send byte[] that you get by using your own serializer ; through
kafka ().On the reciving side u can deseraialize from the byte[] and read
back your object.for using this you will have to
supply serializer.class=kafka.serializer.DefaultEncoder in the properties.
On Tue, May 20, 2014 at
Take a look at Camus https://github.com/linkedin/camus/
François Langelier
Étudiant en génie Logiciel - École de Technologie
Supérieurehttp://www.etsmtl.ca/
Capitaine Club Capra http://capra.etsmtl.ca/
VP-Communication - CS Games http://csgames.org 2014
Jeux de Génie http://www.jdgets.com/ 2011
The customized encoder/decoder has to have a constructor that takes
(VerifiableProperties: props). Alternatively, you could do the
encoding/decoding outside of Kafka client and just send byte[] to Kafka.
The pluggable encoder/decoder will be gradually phased out in the future.
Thanks,
Jun
On
Thanks Pushkar for your response.
I tried to send my own byte array; however the Kafka Producer Class does not
take byte [] as input type. Do you have an example of this? Please share if you
do; really appreciate.
Here is my code:
public class TestEventProducer {
public static void
We plan to work on the feature this summer, and make it available in the
0.9 release. Please try it out then and give us any feedbacks you have.
Guozhang
On Tue, May 20, 2014 at 9:23 AM, Bob Potter bobby.pot...@gmail.com wrote:
Hi Guozhang,
That looks great! I think it would solve our case.
I checked the stats with jconsole and it confirms the reading I've been getting
with jmxtrans so the problem is with the jmx beans themselves I think. The
stats just came back to normal again and I don't know why. I haven't made any
changes to the kafka brokers.
-Original Message-
To be more clear,
1. I am using jmxtrans to get the data not jmstats. Sorry about the misspelling.
2. When I say the stats zero out, I mean that I am not able to get new values
when I refresh with a new query from jmxtrans or jconsole. This only happens
for the
Sorry I lied,
The following do not update:
ProduceRequestsPerSecond
FetchRequestsPerSecond
AvgProduceRequestMs
MaxProduceRequestMs
AvgFetchRequestMs
MaxFetchRequestMs
BytesReadPerSecond
BytesWrittenPerSecond
These stats do update with new values:
NumFetchRequests
NumProduceRequests
It is not recommended to install both kafka and zookeeper on the same box
as both would fight for the available memory and performance will degrade.
Thanks
Neha
On Mon, May 19, 2014 at 7:29 AM, S Ahmed sahmed1...@gmail.com wrote:
Hi,
I like how kafka operates, but I'm wondering if it is
Yes agreed, but I have done some load testing before and kafka was doing
10's of thousands of messages per second.
If I am doing only hundreds, I think it could handle it for now. Like I
said this is small scale.
On Tue, May 20, 2014 at 2:51 PM, Neha Narkhede neha.narkh...@gmail.comwrote:
It
If you really only care about small scale (no HA, no horizontal
scaling), you could also consider using Redis instead of Kafka for
queueing.
- Niek
On Tue, May 20, 2014 at 2:23 PM, S Ahmed sahmed1...@gmail.com wrote:
Yes agreed, but I have done some load testing before and kafka was doing
Hi,
Is there java API in kafka to list topics and partitions in the kafka broker?
Thanks,
Saurabh.
There is a Scala API. You can take a look at TopicCommand.scala as
kafka-topics.sh simply calls that class.
Tim
On Tue, May 20, 2014 at 3:41 PM, Saurabh Agarwal (BLOOMBERG/ 731 LEX
-) sagarwal...@bloomberg.net wrote:
Hi,
Is there java API in kafka to list topics and partitions in the kafka
Hi guys,
So far, is there a way to track the asyn producer callback.
My requirement is basically if all nodes of the topic goes down, can I
pause the producer and after the broker comes back online, continue to
produce from the failure point?
Best,
Siyuan
Thanks. I will look into it.
- Original Message -
From: Timothy Chen tnac...@gmail.com
At: Tuesday, May 20, 2014 18:56
There is a Scala API. You can take a look at TopicCommand.scala as
kafka-topics.sh simply calls that class.
Tim
On Tue, May 20, 2014 at 3:41 PM, Saurabh Agarwal
We introduced callbacks in the new producer. It's only available in trunk
though.
Thanks,
Jun
On Tue, May 20, 2014 at 4:42 PM, hsy...@gmail.com hsy...@gmail.com wrote:
Hi guys,
So far, is there a way to track the asyn producer callback.
My requirement is basically if all nodes of the
You can issue a TopicMetadataRequest. See
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Thanks,
Jun
On Tue, May 20, 2014 at 3:41 PM, Saurabh Agarwal (BLOOMBERG/ 731 LEX -)
sagarwal...@bloomberg.net wrote:
Hi,
Is there java API in kafka to list topics and
Thanks Jun and Francois.
We used Kafka 0.8.0 previously. We got some weird error when expanding
cluster and it couldn't be finished.
Now we use 0.8.1.1, I would have a try on cluster expansion sometime.
I read the discussion on that jira issue and I agree with points raised
there.
HDFS was also
Hangjun,
Does having Kafka in Yarn would be a big architectural change from where
it is now? From what I have seen on most typical setup you want machines
optimized for Kafka, not just it on top of hdfs.
-Steve
On Tue, May 20, 2014 at 8:37 PM, Hangjun Ye yehang...@gmail.com wrote:
Thanks
Hi,
We have several Kafka clusters in production, and we've had to reassign
replication a few times now in production. Some of our topic/partitions
are pretty large, up to 32 partitions per topic, and 16GB per partition, so
adding a new broker and/or repairing a broker that had been down for
Hi Steve,
Yes, what I want is that Kafka doesn't have to care about machines
physically (as an option).
Best,
Hangjun
2014-05-21 11:46 GMT+08:00 Steve Morin st...@stevemorin.com:
Hangjun,
Does having Kafka in Yarn would be a big architectural change from where
it is now? From what I have
ProducerString, byte[] producer = new ProducerString, byte[](config);
Try this.
On Wed, May 21, 2014 at 12:26 AM, Neha Narkhede neha.narkh...@gmail.comwrote:
Pradeep,
If you are writing a POC, I'd suggest you do that using the new producer
APIs
23 matches
Mail list logo
