Re: How to handle the failure of Kafka producers

[Adam Dubiel]

You might consider the approach we are using in Hermes (a pubsub system
with HTTP interface on top of Kafka):

http://hermes-pubsub.readthedocs.io/en/latest/configuration/buffer-persistence/

We use Chronicle Map to persist things that go into Kafka producer into
memory mapped file. In case of process failure, we can be sure that system
takes care of persisting memory contents to disk. On restart it is read and
sent to Kafka. You can see the code here:
https://github.com/allegro/hermes/tree/master/hermes-frontend/src/main/java/pl/allegro/tech/hermes/frontend/buffer/chronicle

BR,
Adam

2016-06-01 18:14 GMT+02:00 Xin Chen :

> Here we have a scenario that we are considering to use Kafka as a
> fault-tolerant message system.
>
> There is an external source that keeps generating data. We are thinking of
> deploying Kafka to store the produced data. It is assumed that the external
> source is safe enough and would not crash or any bad things, since we have
> no control on it. Also we can not deploy something like Kafka producer on
> the source.
> Therefore we are planning to configure the Kafka producer to receive the
> data from the source, and then the brokers serve as the down-streamer of
> the producer.
>
> We understand that the brokers are fault-tolerant by replication. How does
> Kafka handle the failure of the producers? How about the data cached in the
> producer’s accumulator, but not sent to the broker? Do we need to manage
> this inside of producer by ourselves?
>
> Thanks in advance,
> Xin
>

Re: Kafka - Rest api query

[Adam Dubiel]

I don't think you need to write it from scratch, Hermes project
http://hermes-pubsub.readthedocs.org/en/latest/ does this (and more). You
could probably use only consumers module to change pull to push and push
messages from Kafka to other REST services. It has all the retry and send
rate auto adjusting magic you will probably need at some point.

2015-10-20 23:51 GMT+02:00 Kudumula, Surender :

> Actually iam planning to write a consumer in a rest client where kafka
> topic is residing and send the object from rest client to another
> webservice which accepts rest api service.
>
> Regards
>
> Surender Kudumula
> Big Data Consultant - EMEA
> Analytics & Data Management
>
> surender.kudum...@hpe.com
> M +44 7795970923
>
> Hewlett-Packard Enterprise
> Cain Rd,
> Bracknell
> RG12 1HN
> UK
>
>
>
> -Original Message-
> From: Selina Tech [mailto:swucaree...@gmail.com]
> Sent: 20 October 2015 21:50
> To: users@kafka.apache.org
> Subject: Re: Kafka - Rest api query
>
> Could you write a consumer at your rest server?
>
> On Tue, Oct 20, 2015 at 1:18 PM, Kudumula, Surender <
> surender.kudum...@hpe.com> wrote:
>
> > Thanks for the reply. Iam looking to know if its possible to route
> > binary objects messages to rest api service from kafka. If so please let
> me know.
> > Otherwise I can consume the binary object using java consumer and then
> > create a rest client and send the binary message via HTTP POST to rest
> > server. Appreciate your suggestions. Thanks
> >
> > Regards
> >
> > Surender Kudumula
> > Big Data Architect - EMEA
> > Analytics & Data Management
> >
> > surender.kudum...@hp.com
> > M +44 7795970923
> >
> > Hewlett-Packard Company
> > 88 Wood St
> > London EC2V 7QT
> > UK
> >
> >
> >
> > -Original Message-
> > From: Sharninder [mailto:sharnin...@gmail.com]
> > Sent: 20 October 2015 16:35
> > To: users@kafka.apache.org
> > Subject: Re: Kafka - Rest api query
> >
> > Sounds like an app design decision. What help can this list give you ?
> >
> >
> > > On 20-Oct-2015, at 8:07 PM, Kudumula, Surender <
> > surender.kudum...@hpe.com> wrote:
> > >
> > > Dear sir/madam
> > > I have a query. We are working on POC at the moment and we are using
> > kafka to produce and consume messages. I have one component which
> > consumes the request from topic and processes it and creates a file
> > and again produce the java object as byte array to another kafka
> > topic. Now I have two options
> > > 1. Is to write a java rest client and send the bytearray java
> object
> > to rest api service in another component
> > > 2. Or I was wondering if its possible to route the message from
> > kafka topic to another component rest APi server.
> > >
> > > Please let me know we are using kafka 0.8 and new producer. Any
> > suggestions would be appreciated?
> > >
> > > Regards
> > >
> > > Surender Kudumula
> > > Big Data Consultant - EMEA
> > > Analytics & Data Management
> > >
> > > surender.kudum...@hpe.com
> > > M +44 7795970923
> > >
> > > Hewlett-Packard Enterprise
> > > Cain Rd,
> > > Bracknell
> > > RG12 1HN
> > > UK
> > >
> > >
> > >
> >
>

Re: Custom Zookeeper install with kafka

[Adam Dubiel]

Hi,

I don't think it matters much which version of ZK will you use (meaning
minor/patch versions). We have been using 3.4.6 for some time and it works
flawlessly.

BR,
Adam

2015-07-22 18:40 GMT+02:00 Prabhjot Bharaj prabhbha...@gmail.com:

 Hi,

 I've read on the Kafka documentation page that the zookeeper version used
 is 3.3.4

 However, at my work, I've noticed certain problems with v3.3.4 (and the
 problems are documented here: http://zookeeper.apache.org/releases.html

 The latest stable version of zookeeper is 3.4.6 and has a lot of bug fixes
 from 3.3.4

 Has anyone tried using this version of zookeeper with kafka 0.8.2.1 ?

 Regards,
 Prabhjot

Re: Issue with corrupt message in Topic

[Adam Dubiel]

Hi Nicolas,

From my experience there are only two ways out:
1) wait for retention time to pass, so data gets deleted (this is usually
unacceptable)
2) trace offset of corrupt message on all affected subscriptions and skip
this message by overwriting it (offset+1)

Problem is, that when encountering corrupt message, high level consumer
iterator goes into invalid state and closes. There is no way to skip this
message or recover from it without skipping offsets. You might try to use
SimpleConsumer though. Maybe someone knows other ways to deal with this
problem, but we haven't found any.

BR,
Adam

2015-07-21 9:38 GMT+02:00 Nicolas Phung nicolas.ph...@gmail.com:

 Hello,

 I'm using Confluent Kafka (0.8.2.0-cp). When I'm trying to process message
 from my Kafka topic with Spark Streaming, I've got the following error :

 kafka.message.InvalidMessageException: Message is corrupt (stored crc =
 3561357254, computed crc = 171652633)
 at kafka.message.Message.ensureValid(Message.scala:166)
 at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.
 scala:102)
 at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.
 scala:33)
 at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTempla
 te.scala:66)
 at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
 at org.apache.spark.streaming.kafka.ReliableKafkaReceiver$Messa
 geHandler.run(ReliableKafkaReceiver.scala:265)
 at java.util.concurrent.Executors$RunnableAdapter.call(
 Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
 Executor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
 lExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 15/07/20 15:56:57 INFO BlockManager: Removing broadcast 4641
 15/07/20 15:56:57 ERROR ReliableKafkaReceiver: Error handling message
 java.lang.IllegalStateException: Iterator is in failed state
 at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:54)
 at org.apache.spark.streaming.kafka.ReliableKafkaReceiver$Messa
 geHandler.run(ReliableKafkaReceiver.scala:265)
 at java.util.concurrent.Executors$RunnableAdapter.call(
 Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
 Executor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
 lExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

 From my understanding, there's some corrupt message in my topic. I'm using
 the new Producer API to send message compress with Snappy. I found an old
 topic talking about it but with no further step to resolve the issue. Do
 you have any informations regarding this ?

 Is it possible in Kafka to somehow reread the topic and drop corrupt
 message ?

 Regards,
 Nicolas PHUNG

Re: How to monitor Kafka offset

[Adam Dubiel]

As i mentioned earlier, this feature has not yet been released, but out
pull-request has been approved by Quantifind. Here it is:

https://github.com/quantifind/KafkaOffsetMonitor/pull/58/files

If you would like to use it now, you would have to build Offset Monitor jar
yourself using code from fork. Otherwise you will need to wait for
Quantifind to merge  release new version.

This is repository of Graphite integration plugin:

https://github.com/allegro/kafka-offset-monitor-graphite

We are already using it on production, works great and gives us tons of
data to monitor.

Having data in Graphite, you can use some Graphite monitoring tool like
Cabot (https://github.com/arachnys/cabot), that triggers alerts based on
stuff from Graphite.

Adam


2015-07-10 10:45 GMT+02:00 Anandh Kumar anandhz...@gmail.com:

 Adam,

 Tried to config KafkaOffsetMonitor it is working fine.

 But how can we  integrated with Graphite to add alerting ? can please
 explain in details and if you have any doc can you please provide?

 Thanks  Regards,
 -Anandh Kumar

 On Fri, Jul 10, 2015 at 12:58 PM, Adam Dubiel dubiel.a...@gmail.com
 wrote:

  We are using kafka offset monitor (
  http://quantifind.com/KafkaOffsetMonitor/),
  which we recently integrated with Graphite to add alerting and better
  graphing - it should be accessible in newest version, not yet released.
 It
  works only with ZK offsets though.
 
  2015-07-10 9:24 GMT+02:00 Anandh Kumar anandhz...@gmail.com:
 
   Thanks Rahul for your reply
  
   On Fri, Jul 10, 2015 at 11:11 AM, Rahul Jain rahul...@gmail.com
 wrote:
  
Burrow works only if you are storing the offsets in kafka topic, not
zookeeper. You can also take a look at Kafka web console ( it has a
   memory
leak bug but a patch is available ).
On 10 Jul 2015 09:34, Jiangjie Qin j...@linkedin.com.invalid
  wrote:
   
 You can take a look at Burrow. We use it in LinkedIn.

 Thanks,

 Jiangjie (Becket) Qin

 On 7/9/15, 8:53 PM, Anandh Kumar anandhz...@gmail.com wrote:

 Hi
 
 We have any monitor tool which monitor kafka offset?
 
 I need some opensource admin tool for kafka.
 
 Please guide me.
 
 Regards,
 -Anandh Kumar


   
  
 

Re: How to monitor Kafka offset

[Adam Dubiel]

We are using kafka offset monitor (http://quantifind.com/KafkaOffsetMonitor/),
which we recently integrated with Graphite to add alerting and better
graphing - it should be accessible in newest version, not yet released. It
works only with ZK offsets though.

2015-07-10 9:24 GMT+02:00 Anandh Kumar anandhz...@gmail.com:

 Thanks Rahul for your reply

 On Fri, Jul 10, 2015 at 11:11 AM, Rahul Jain rahul...@gmail.com wrote:

  Burrow works only if you are storing the offsets in kafka topic, not
  zookeeper. You can also take a look at Kafka web console ( it has a
 memory
  leak bug but a patch is available ).
  On 10 Jul 2015 09:34, Jiangjie Qin j...@linkedin.com.invalid wrote:
 
   You can take a look at Burrow. We use it in LinkedIn.
  
   Thanks,
  
   Jiangjie (Becket) Qin
  
   On 7/9/15, 8:53 PM, Anandh Kumar anandhz...@gmail.com wrote:
  
   Hi
   
   We have any monitor tool which monitor kafka offset?
   
   I need some opensource admin tool for kafka.
   
   Please guide me.
   
   Regards,
   -Anandh Kumar
  
  
 

Re: querying messages based on timestamps

[Adam Dubiel]

We faced similar problem and ended up with implementing variant of golden
section search, that reads message using simple consumer and checks the
timestamp (timestamps are appended by our producer though, they do not come
from any Kafka metadata) till it finds message closest to given date.

Adam

2015-06-30 21:52 GMT+02:00 Shushant Arora shushantaror...@gmail.com:

 Is it possible using low level consumer to get kafka messages based on
 timestamp, say I want to get all messages from last 5 minutes. I don't know
 what were offsets of partitions 5 minutes back.

 In low level consumer : when I gave epoch for  whichTime , it failed.
  requestInfo.put(topicAndPartition, new
 PartitionOffsetRequestInfo(whichTime,
 1));

 Is only latest and earliest supported in timestamp,Is there any way to
 filter messages based on timestamp?

 Thanks
 Shushant

Re: Manual Offset Commits with High Level Consumer skipping messages

[Adam Dubiel]

Hi,

We have been solving this very problem in Hermes. You can see what we came
up by examining classes located here:

https://github.com/allegro/hermes/tree/master/hermes-consumers/src/main/java/pl/allegro/tech/hermes/consumers/consumer/offset

We are quite sure this gives us at-least-once guarantees. This is basically
a queue of offsets per partition, each with ready/not ready marker. They
are sorted, and every N seconds we try to commit highest offsets, whose all
predecessors are ready (per partition). Since there is possibility of never
committing (i.e. one message gets stuck in processing and it never reports
back) we have some time-based monitoring that alerts us if there was no
commit for this partition for a long time. We never do a force-commit,
since we value our messages a lot :)

This is internal code, so it's not pure, as it references our configs and
metrics, but i hope you can get the idea :)


2015-06-19 16:07 GMT+02:00 Carl Heymann ch.heym...@gmail.com:

 From my understanding of the code (admittedly very limited), the offset in
 OffsetAndMetadata corresponds to the start of the message just obtained
 from iterator.next(). So if you commit that, a restarted consumer should
 get that message again. So it should actually continue at the previous
 message relative to what you might think. You should also be able to commit
 that offset even before processing the message, and still get at-least once
 delivery. So you should not miss messages, just get some duplicate
 deliveries.

 You mention some issues with produced messages being stored on the
 partitions. Is this an independent problem from the consumer issue?

 Do you have the full test code somewhere? I'm also trying to get
 at-least-once delivery with the high level consumer.

 On Fri, Jun 19, 2015 at 2:07 PM, noah iamn...@gmail.com wrote:

  It is the value we get from calling MessageAndMetadata#offset() for the
  last message processed. The MessageAndMetadata instance comes from the
  ConsumerIterator.
 
  On Fri, Jun 19, 2015 at 2:31 AM Carl Heymann ch.heym...@gmail.com
 wrote:
 
   How are you tracking the offsets that you manually commit? I.e. where
 do
   you get the metadata for the consumed messages?
  
   On Thu, Jun 18, 2015 at 11:21 PM, noah iamn...@gmail.com wrote:
  
We are in a situation where we need at least once delivery. We have a
thread that pulls messages off the consumer, puts them in a queue
 where
they go through a few async steps, and then after the final step, we
  want
to commit the offset to the messages we have completed. There may be
   items
we have not completed still being processed, so
consumerConnector.commitOffsets() isn't an option for us.
   
We are manually committing offsets to Kafka (0.8.2.1) (auto commit is
   off.)
   
We have a simple test case that is supposed to verify that we don't
  lose
any messages if the Kafka server is shut down:
   
// there are 25 messages, we send a few now and a few after the
 server
comes back up
for (TestMessageClass mess : messages.subList(0, mid)) {
producer.send(mess);
}
   
stopKafka(); // in memory KafkaServer
startKafka();
   
for (TestMessageClass mess : messages.subList(mid, total)) {
producer.send(mess);
}
   
int tries = 0;
while(testConsumer.received.size()  total  tries++  10) {
Thread.sleep(200);
}
assertEquals(keys(testConsumer.received),
keys(ImmutableSet.copyOf(messages)));
   
The test consumer is very simple:
   
ConsumerIterator iterator;
while(iterator.hasNext()) {
process(iterator.next());
}
   
// end of process:
   commit(messageAndMetadata.offset());
   
commit is basically the commit code from this page:
   
   
  
 
 https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
,
but runs the commit in a separate thread so it wont interfere with
 the
consumer.
   
Here is the strange thing: If we do not commit, the test passes every
   time.
Kafka comes back up and the high level consumer picks up right where
 it
left off. But if we do commit, it does not recover, or we lose
  messages.
With 1 partition, we only get some prefix of the messages produced
  before
stopKafka(). With 2, one of the partitions never gets any of the
  messages
sent in the second half, while the other gets a prefix, but not all
 of
   the
messages for that partition.
   
It seems like the most likely thing is that we are committing the
 wrong
offsets, but I cannot figure out how that is happening. Does the
 offset
   in
MessageAndMetadata not correspond to the offset in OffsetAndMetadata?
   
Or do we have to abandon the high level consumer entirely if we want
 to
manually commit in this way?
   
  
 

Re: Increased replication factor. Replication didn't happen!

[Adam Dubiel]

I just tried it out on my 0.8.2 cluster and it worked just fine - the ISR
grew, replica factor changed and data was physically moved to new brokers.
Was there not output/no logs? I see things like

INFO Created log for partition [topicName,7] in /opt/kafka/ with
properties {.. some json}

in server.log on new brokers.

2015-06-11 21:02 GMT+02:00 Dillian Murphey crackshotm...@gmail.com:

  Oh, hmm.  There is even documentation on it:


 http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor

 Having a difficult time sifting through the logs.  Is this not a common
 operation?  Do users normally just delete the topic and create a new one??



 On Wed, Jun 10, 2015 at 8:20 PM, Shady Xu shad...@gmail.com wrote:

  Right now, Kafka topics do not support changing replication factor or
  partition number after creation. The  kafka-reassign-partitions.sh tool
 can
  only reassign existent partitions.
 
  2015-06-11 9:31 GMT+08:00 Gwen Shapira gshap...@cloudera.com:
 
   What do the logs show?
  
   On Wed, Jun 10, 2015 at 5:07 PM, Dillian Murphey
   crackshotm...@gmail.com wrote:
Ran this:
   
$KAFKA_HOME/bin/kafka-reassign-partitions.sh
   
But Kafka did not actually do the replication. Topic description
 shows
   the
right numbers, but it just didn't replicate.
   
What's wrong, and how do I trigger the replication to occur??
   
I'm running 0.8.2.0
   
thanks
  
 

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

[Adam Dubiel]

Thanks for the feedback. We will improve our documentation to make it clear
what guarantees we concentrate on and what are the tradeoffs. And think of
some catchy way to describe the system :)

2015-05-25 10:02 GMT+02:00 Daniel Compton daniel.compton.li...@gmail.com:

 Hi Adam

 No problem, I'm glad you had a good break :)

 I think the goals and tradeoffs that Hermes chose provide a useful system,
 I'm just not sure reliable is quite the right word to use when describing
 it. When I think of 'reliable' in the context of a message queue I think of
 guaranteed delivery once the message is accepted (subject to some
 limitations like a quorum of nodes staying up).

 I get the other meaning of reliable that you're using - delivered within a
 predictable time frame, but I wonder if something more like predictable,
 bounded response times would be more accurate (perhaps there's a catchier
 way of saying it :))

 Documenting the tradeoffs and failure scenarios will make it easier for
 people to understand whether it fits their needs. I can imagine that you
 could couple Hermes with an offline reconciliation to check any 201'd
 messages were successfully delivered to Kafka.

 Anyway, great work on this and thanks for sharing!
 On Mon, 25 May 2015 at 1:26 am Adam Dubiel dubiel.a...@gmail.com wrote:

  Hi Daniel,
 
  First of all sorry for late response, i enjoyed short vacation :)
 
  I guess the documentation might be bit misleading here, and so we should
  improve it: we do not aim (and can't) provide higher guarantees than
 Kafka.
 
  We want to be as bullteproofs as possible in REST interface segments. In
  our SLA we concentrate a lot on REST availability and response time. We
 can
  also withstand some short-term Kafka outages. Still, the only goal of
  Hermes frontend is to push event to Kafka, as only this way we can assure
  our customers that the event will be delivered (and is stored reliably).
  Thus, we do not plan on making any distributed storage here - Kafka is
 our
  storage.
 
  Adam
 
 
 
  2015-05-20 11:49 GMT+02:00 Daniel Compton 
 daniel.compton.li...@gmail.com
  :
 
   Hi Adam
  
   Firstly, thanks for open sourcing this, it looks like a great tool and
 I
   can imagine a lot of people will find it very useful.
  
   I had a few thoughts reading the docs. I may have misunderstood things
  but
   it seems that your goal of meeting a strict SLA conflicts with your
 goal
  of
   bulletproof delivery. Even if you have a durable queue on disk, when
 you
   send a 202 Accepted you could still lose messages if you lost the disk
   storing the data.
  
   I realise this has a small chance of happening, but I don't think you
 can
   guarantee bulletproof delivery if only a single server stores the
 message
   while its in transit before being accepted by Kafka.
  
   Could you expand on the reliability guarantees you're looking to offer
  and
   how they can be stronger than the ones provided by Kafka?
  
   Thanks, Daniel.
   On Tue, 19 May 2015 at 2:57 am Adam Dubiel dubiel.a...@gmail.com
  wrote:
  
Hello everyone,
   
I'm technical team lead of Hermes project. I will try to answer
 already
posted questions, but feel free to ask me anything.
   
1) Can you comment on how this compares to Confluent's REST proxy?
   
We do not perceive Hermes as mere proxy. While Confluent product
 wants
  to
help services written in non-jvm languages in connecting to Kafka,
  Hermes
is more then that. First of all we wanted to make connecting to
 PubSub
  as
easy as possible, hence REST API for publishing (this is same as REST
proxy), but also converting from pull to push on consumer side. This
   means,
that our consumers don't need to include Kafka drivers, handle
 retries
  or
worry about linear commit offset. Instead, our hermes-consumer module
implements retry strategies and adaptive output-rate algorithm.
  Consumer
needs to register HTTP (or JMS) endpoint that can accept POST message
  and
that is all.
   
Secondly, on the publisher side, we wanted to create somewhat
   bullet-proof
solution capable of accepting very sensitive data. This means we use
   Kafka
producer buffer to store unsent messages (our internal solution also
persists this buffer to disk, this will be ported to OpenSource soon)
  and
we can guarantee maximum response time (SLA). We also are able to use
different levels of Kafka ACKs per topic (-1 or 1 currently).
   
Last but not least, we mitigate some of administrative issues: added
  tons
of metrics that can be reported to graphite, message state tracking
 for
debugging purposes, easy to use REST API for previewing messages
 stored
   at
Kafka or to retransmit events starting from given timestamp (not
   offset!),
   
   
2) Performance
   
We plan on making tests public, but they are not there yet. The
 numbers
   in
docs come from our production environment, but they should

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

[Adam Dubiel]

Hi Daniel,

First of all sorry for late response, i enjoyed short vacation :)

I guess the documentation might be bit misleading here, and so we should
improve it: we do not aim (and can't) provide higher guarantees than Kafka.

We want to be as bullteproofs as possible in REST interface segments. In
our SLA we concentrate a lot on REST availability and response time. We can
also withstand some short-term Kafka outages. Still, the only goal of
Hermes frontend is to push event to Kafka, as only this way we can assure
our customers that the event will be delivered (and is stored reliably).
Thus, we do not plan on making any distributed storage here - Kafka is our
storage.

Adam



2015-05-20 11:49 GMT+02:00 Daniel Compton daniel.compton.li...@gmail.com:

 Hi Adam

 Firstly, thanks for open sourcing this, it looks like a great tool and I
 can imagine a lot of people will find it very useful.

 I had a few thoughts reading the docs. I may have misunderstood things but
 it seems that your goal of meeting a strict SLA conflicts with your goal of
 bulletproof delivery. Even if you have a durable queue on disk, when you
 send a 202 Accepted you could still lose messages if you lost the disk
 storing the data.

 I realise this has a small chance of happening, but I don't think you can
 guarantee bulletproof delivery if only a single server stores the message
 while its in transit before being accepted by Kafka.

 Could you expand on the reliability guarantees you're looking to offer and
 how they can be stronger than the ones provided by Kafka?

 Thanks, Daniel.
 On Tue, 19 May 2015 at 2:57 am Adam Dubiel dubiel.a...@gmail.com wrote:

  Hello everyone,
 
  I'm technical team lead of Hermes project. I will try to answer already
  posted questions, but feel free to ask me anything.
 
  1) Can you comment on how this compares to Confluent's REST proxy?
 
  We do not perceive Hermes as mere proxy. While Confluent product wants to
  help services written in non-jvm languages in connecting to Kafka, Hermes
  is more then that. First of all we wanted to make connecting to PubSub as
  easy as possible, hence REST API for publishing (this is same as REST
  proxy), but also converting from pull to push on consumer side. This
 means,
  that our consumers don't need to include Kafka drivers, handle retries or
  worry about linear commit offset. Instead, our hermes-consumer module
  implements retry strategies and adaptive output-rate algorithm. Consumer
  needs to register HTTP (or JMS) endpoint that can accept POST message and
  that is all.
 
  Secondly, on the publisher side, we wanted to create somewhat
 bullet-proof
  solution capable of accepting very sensitive data. This means we use
 Kafka
  producer buffer to store unsent messages (our internal solution also
  persists this buffer to disk, this will be ported to OpenSource soon) and
  we can guarantee maximum response time (SLA). We also are able to use
  different levels of Kafka ACKs per topic (-1 or 1 currently).
 
  Last but not least, we mitigate some of administrative issues: added tons
  of metrics that can be reported to graphite, message state tracking for
  debugging purposes, easy to use REST API for previewing messages stored
 at
  Kafka or to retransmit events starting from given timestamp (not
 offset!),
 
 
  2) Performance
 
  We plan on making tests public, but they are not there yet. The numbers
 in
  docs come from our production environment, but they should be taken with
  grain of salt. Hermes performance depends highly on underlying Kafka
  cluster performance. Our cluster is deployed in cloud (on SSDs), bare
 metal
  deployments would probably achieve a lot better performance. Still, the
  most important metric here is not total response time, but Hermes
 overhead
  over pure Kafka. Looks like this is neglible in our cloud deployment
 (p99 
  0.2ms), but we will be crunching those numbers and publish them in docs.
 
  3) Topics/subscriptions limit
 
  We are limited by Kafka as well, though we never encountered any problems
  with this (still, we have only 100-150 topics). We want to scale out by
  making Hermes multi-kafka aware (in effort to become multi DC). Currently
  management node can manage multiple Kafka clusters, but as soon as we
  deploy it on production we will add some more documentation on
 architecture
  and deployment.
 
 
  We should create FAQ that would answer some most popular questions.
 
 
  2015-05-18 13:14 GMT+02:00 Marcin Kuthan marcin.kut...@gmail.com:
 
   Hi Warren
  
   With Hermes, the publisher is never blocked. Even if message has not
   been sent to Kafka cluster, or if message has been sent but not
   acknowledged. It is especially useful if your system needs to have
   strict SLA guarantees.
  
   From the consumers perspective there is retrying policy if the
   consumer is not able to handle published message. In addition, Hermes
   is able to adjust speed of sending messages to actual situation (i.e

Re: Consumers in a different datacenters

[Adam Dubiel]

Hi Bill,

I don't know if this is exactly the same case (last part when they get the
topic tehy apply locally is bit unclear), but we have setup with Kafka in
DC A and consumers both in DC A and DC B. Actually we also have producers
in A and B writing to Kafka in A, but we are trying to change this (we have
7ms trip time between DCs). We want to achieve DC-locality by using
DC-aware service discovery and Hermes, our product built on top of Kafka.

BR,
Adam

2015-05-19 20:46 GMT+02:00 Bill Hastings bllhasti...@gmail.com:

 Hi All

 Has anyone tried this? We have two data centers A and B. We would like data
 replicated between A and B. So I would like to have a kafka cluster set up
 in A and B. When we need to replicate from A--B I would like the app in A
 publish a topic to the kafla cluster in data center A. The corresponding
 consumers are in data center B. When they get the topic they apply locally.
 Does this sound like a viable solution?

 --
 Cheers
 Bill

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

[Adam Dubiel]

Hello everyone,

I'm technical team lead of Hermes project. I will try to answer already
posted questions, but feel free to ask me anything.

1) Can you comment on how this compares to Confluent's REST proxy?

We do not perceive Hermes as mere proxy. While Confluent product wants to
help services written in non-jvm languages in connecting to Kafka, Hermes
is more then that. First of all we wanted to make connecting to PubSub as
easy as possible, hence REST API for publishing (this is same as REST
proxy), but also converting from pull to push on consumer side. This means,
that our consumers don't need to include Kafka drivers, handle retries or
worry about linear commit offset. Instead, our hermes-consumer module
implements retry strategies and adaptive output-rate algorithm. Consumer
needs to register HTTP (or JMS) endpoint that can accept POST message and
that is all.

Secondly, on the publisher side, we wanted to create somewhat bullet-proof
solution capable of accepting very sensitive data. This means we use Kafka
producer buffer to store unsent messages (our internal solution also
persists this buffer to disk, this will be ported to OpenSource soon) and
we can guarantee maximum response time (SLA). We also are able to use
different levels of Kafka ACKs per topic (-1 or 1 currently).

Last but not least, we mitigate some of administrative issues: added tons
of metrics that can be reported to graphite, message state tracking for
debugging purposes, easy to use REST API for previewing messages stored at
Kafka or to retransmit events starting from given timestamp (not offset!),


2) Performance

We plan on making tests public, but they are not there yet. The numbers in
docs come from our production environment, but they should be taken with
grain of salt. Hermes performance depends highly on underlying Kafka
cluster performance. Our cluster is deployed in cloud (on SSDs), bare metal
deployments would probably achieve a lot better performance. Still, the
most important metric here is not total response time, but Hermes overhead
over pure Kafka. Looks like this is neglible in our cloud deployment (p99 
0.2ms), but we will be crunching those numbers and publish them in docs.

3) Topics/subscriptions limit

We are limited by Kafka as well, though we never encountered any problems
with this (still, we have only 100-150 topics). We want to scale out by
making Hermes multi-kafka aware (in effort to become multi DC). Currently
management node can manage multiple Kafka clusters, but as soon as we
deploy it on production we will add some more documentation on architecture
and deployment.


We should create FAQ that would answer some most popular questions.


2015-05-18 13:14 GMT+02:00 Marcin Kuthan marcin.kut...@gmail.com:

 Hi Warren

 With Hermes, the publisher is never blocked. Even if message has not
 been sent to Kafka cluster, or if message has been sent but not
 acknowledged. It is especially useful if your system needs to have
 strict SLA guarantees.

 From the consumers perspective there is retrying policy if the
 consumer is not able to handle published message. In addition, Hermes
 is able to adjust speed of sending messages to actual situation (i.e.
 subscribing service has slown down due to heavy GC)


 On 18 May 2015 at 09:56, Warren Henning warren.henn...@gmail.com wrote:
  Interesting. Thanks for sharing and working on this!
 
  Can you comment on how this compares to Confluent's REST proxy?
 
  On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan marcin.kut...@gmail.com
 
  wrote:
 
  Hi Everyone
 
  Hermes is an asynchronous message broker built on top of Kafka. It
  provides reliable, fault tolerant REST interface for message
  publishing and adaptive push mechanisms for message sending. Hermes is
  used as a message broker for asynchronous communication between
  microservices.
 
  Some of the main features:
 
  - Performance and scalability - Hermes in production handling up to
  30.000 msgs/sec with 99.9th percentile latency below 100 ms on a
  4-node cluster.
 
  - Reliability - Hermes is used for publishing sensitive data, such as
  billing events, user account changes etc. Hermes allows to define more
  reliable policy for those important events - require acknowledge from
  all Kafka replicas and increase request timeouts.
 
  - Push model - It makes receiving messages from Hermes dead simple:
  you just have to write one HTTP endpoint in your service. It’s up to
  Hermes to create Kafka consumer, redeliver messages, keep eye on
  throughput limits etc.
 
  Feedback and comments are welcome, you can find Hermes documentation at:
  http://hermes-pubsub.readthedocs.org/en/latest/index.html
 
  Hermes is published under Apache Licence:
  https://github.com/allegro/hermes
 
  Best Regards,
  Marcin