You might consider the approach we are using in Hermes (a pubsub system
with HTTP interface on top of Kafka):
http://hermes-pubsub.readthedocs.io/en/latest/configuration/buffer-persistence/
We use Chronicle Map to persist things that go into Kafka producer into
memory mapped file. In case of
I don't think you need to write it from scratch, Hermes project
http://hermes-pubsub.readthedocs.org/en/latest/ does this (and more). You
could probably use only consumers module to change pull to push and push
messages from Kafka to other REST services. It has all the retry and send
rate auto
Hi,
I don't think it matters much which version of ZK will you use (meaning
minor/patch versions). We have been using 3.4.6 for some time and it works
flawlessly.
BR,
Adam
2015-07-22 18:40 GMT+02:00 Prabhjot Bharaj prabhbha...@gmail.com:
Hi,
I've read on the Kafka documentation page that
Hi Nicolas,
From my experience there are only two ways out:
1) wait for retention time to pass, so data gets deleted (this is usually
unacceptable)
2) trace offset of corrupt message on all affected subscriptions and skip
this message by overwriting it (offset+1)
Problem is, that when
integrated with Graphite to add alerting ? can please
explain in details and if you have any doc can you please provide?
Thanks Regards,
-Anandh Kumar
On Fri, Jul 10, 2015 at 12:58 PM, Adam Dubiel dubiel.a...@gmail.com
wrote:
We are using kafka offset monitor (
http://quantifind.com
We are using kafka offset monitor (http://quantifind.com/KafkaOffsetMonitor/),
which we recently integrated with Graphite to add alerting and better
graphing - it should be accessible in newest version, not yet released. It
works only with ZK offsets though.
2015-07-10 9:24 GMT+02:00 Anandh Kumar
We faced similar problem and ended up with implementing variant of golden
section search, that reads message using simple consumer and checks the
timestamp (timestamps are appended by our producer though, they do not come
from any Kafka metadata) till it finds message closest to given date.
Adam
Hi,
We have been solving this very problem in Hermes. You can see what we came
up by examining classes located here:
https://github.com/allegro/hermes/tree/master/hermes-consumers/src/main/java/pl/allegro/tech/hermes/consumers/consumer/offset
We are quite sure this gives us at-least-once
I just tried it out on my 0.8.2 cluster and it worked just fine - the ISR
grew, replica factor changed and data was physically moved to new brokers.
Was there not output/no logs? I see things like
INFO Created log for partition [topicName,7] in /opt/kafka/ with
properties {.. some json}
in
, great work on this and thanks for sharing!
On Mon, 25 May 2015 at 1:26 am Adam Dubiel dubiel.a...@gmail.com wrote:
Hi Daniel,
First of all sorry for late response, i enjoyed short vacation :)
I guess the documentation might be bit misleading here, and so we should
improve it: we do
and
how they can be stronger than the ones provided by Kafka?
Thanks, Daniel.
On Tue, 19 May 2015 at 2:57 am Adam Dubiel dubiel.a...@gmail.com wrote:
Hello everyone,
I'm technical team lead of Hermes project. I will try to answer already
posted questions, but feel free to ask me
Hi Bill,
I don't know if this is exactly the same case (last part when they get the
topic tehy apply locally is bit unclear), but we have setup with Kafka in
DC A and consumers both in DC A and DC B. Actually we also have producers
in A and B writing to Kafka in A, but we are trying to change
Hello everyone,
I'm technical team lead of Hermes project. I will try to answer already
posted questions, but feel free to ask me anything.
1) Can you comment on how this compares to Confluent's REST proxy?
We do not perceive Hermes as mere proxy. While Confluent product wants to
help services
13 matches
Mail list logo
