Re: Simple KafkaProducer to handle multiple requests or not

2017-03-14 Thread Amit K
using too much memory. > > On Mon, Mar 13, 2017 at 5:05 AM, Amit K <amitk@gmail.com> wrote: > > > Hi, > > > > I am using simple kafka producer (java based, version 0.9.0.0) in an > > application where I receive lot of hits (about 50 per seconds, in much > li

Simple KafkaProducer to handle multiple requests or not

2017-03-13 Thread Amit K
Hi, I am using simple kafka producer (java based, version 0.9.0.0) in an application where I receive lot of hits (about 50 per seconds, in much like servlet way) on application that has kafka producer. Per request comes different set of records. I am using only one instance of kafka producer to

Re: Processing older records Kafka Consumer

2016-12-06 Thread Amit K
mes...@gmail.com> wrote: > Do you use auto-commit or committing your self? I'm trying to figure out > how the offset moved if it was stuck. > > On Tue, Dec 6, 2016 at 10:28 AM Amit K <amitk@gmail.com> wrote: > > > Hi, > > > > Is there any way to r

Processing older records Kafka Consumer

2016-12-06 Thread Amit K
Hi, Is there any way to re-consume the older records from Kafka broker with kafka consumer? I am using kafka 0.9.0.0 In one of the scenario, I saw records for 2 days from today were not consumed as consumer was stuck. When the consumer restarted, it started processing records from today but

Re: Re-consume messages

2016-11-07 Thread Amit K
(Becket) Qin > > On Sun, Nov 6, 2016 at 1:30 AM, Amit K <amitk@gmail.com> wrote: > > > Hi All, > > > > I am using kafka 0.9.0.1 with high level java producer and consumer. > > I need to handle a case wherein I need to re-consume the alre

Re-consume messages

2016-11-06 Thread Amit K
Hi All, I am using kafka 0.9.0.1 with high level java producer and consumer. I need to handle a case wherein I need to re-consume the already consumed (and processed) messages say for last 5 days (configurable). Is there any way of achieving the same apart from identifying the offsets for the

Re: Kafka java consumer processes duplicate messages

2016-08-02 Thread Amit K
else? > On Aug 1, 2016 10:59 PM, "Amit K" <amitk@gmail.com> wrote: > > > Thanks for reply, > > > > On producer side, I have ACK as all, with 3 retries, rest all are mostly > > default properties. > > > > With replication factor of 2, I

Kafka java consumer processes duplicate messages

2016-08-01 Thread Amit K
Hi, I am kind of new to Kafka. I have set up a 3 node kafka (1 broker per machine) cluster with 3 node zookeer cluster. I am using Kafka 0.9.0.0 version. The set up works fine wherein from my single producer I am pushing a JSON string to Kafka to a topic with 3 partitions and replication factor

Regarding kafka partition and replication

2016-07-18 Thread Amit K
Hi, I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 zookeeper cluster. So total 9 brokers spread across 3 different machines. I am adhered to Kafka 0.9. In order to optimally use the infrastructure for 2 topics (as of now, is not expected to grow drastically in near future), I am