invoking kafka consumer as soon as message arrives in topic

Hi all Iam trying to write a web application which is invoked when the message arrives in topic. The code is waiting for a message in kafka consumer in while loop and sometimes it picks up the message and sometimes its waiting forever even when the message is produced in the topic. I am

Re: Queue Full

You can try to set "queue.buffering.max.messages" larger, for example: confParam=["queue.buffering.max.messages=200", "batch.num.messages=1000"] And reference for you: https://github.com/edenhill/librdkafka/issues/210 2015-10-27 1:19 GMT+08:00 Prabhjot Bharaj : > Hi,

Regarding the Kafka offset management issue in Direct Stream Approach.

Hi All, We are working in Apache spark with Kafka integration, in this use case we are using DirectStream approach. we want to avoid the data loss in this approach for actually we take offsets and saving that offset into MongoDB. We want some clarification is Spark stores any offsets

Kafka Best Practices in AWS

We installed four nodes (r3.xlarge) with approx 100 topics. Some topics have a replication factor of 4, some have 3 and some have 2. Each node is running about 90% cpu usage and we are seeing performance issues with messages consumption. Would anyone have any words of wisdom or recommendations

Re: Regarding the Kafka offset management issue in Direct Stream Approach.

Questions about spark's kafka integration should probably be directed to the spark user mailing list, not this one. I don't monitor kafka mailing lists as closely, for instance. For the direct stream, Spark doesn't keep any state regarding offsets, unless you enable checkpointing. Have you read

Queue Full

Hi Guys, How could I solving this problem? % Failed to produce message: Local: Queue full % Failed to produce message: Local: Queue full Thanks -- Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Queue Full

Hi Magnus I think this answer c) producing messages at a higher rate than the network or broker can handle How could I manager this? > On 26 Oct 2015, at 17:45, Magnus Edenhill wrote: > > c) producing messages at a higher rate than the network or broker can > handle --

Re: Queue Full

Hi, This is a type of problem where you operating more than network capacity. This can be handled at two places (you decide whichever is useful/practical for use case) :- 1. In case the bottleneck is because of the broker slowness, increase the number of partitions of your topic, balance them

Re: Queue Full

That's librdkafka's producer queue that has run full because: a) client is not connected to broker b) no leader broker for relevant partitions c) producing messages at a higher rate than the network or broker can handle /Magnus 2015-10-26 16:49 GMT+01:00 Eduardo Costa Alfaia

Re: Kafka Best Practices in AWS

Depending on the consumer, I tend to work with c4 types and have found better success. Although we run about 200+ consumers per server on 16 core servers and see moderate load. YMMV On Oct 26, 2015 8:32 AM, "Jennifer Fountain" wrote: > We installed four nodes (r3.xlarge)