Re: Custom AuthenticateCallbackHandler

No I want to check plain user credentials for authentication against an external system. Where best to plug in this check. I thought the new AuthenticateCallbackHandler would be convenient, but I am very much in doubt if this is the right way. It is also not clear for me from the docs how to

Re: Kafka consumer loop exception handling

You should set the reset to latest, commit offsets manually using a rebalance listener. In this way upon seek() you should get all data right. Also when you say “Uncommitted” offset, that means you haven’t really processed them. So you should determine such failure, manually control offset

Re: Custom AuthenticateCallbackHandler

are you referring to onCompleteCallback from https://github.com/apache/kafka/commit/837f31dd1850b179918f83338b4b4487486b2c58 [https://avatars2.githubusercontent.com/u/14958864?s=200=4] KAFKA-6927; Chunked

Re: Is there expiration for committed Offset in the partition

It is an know issue. You can increase the retention time for stored offsets via configs thought. There is already an open PR to fix this issue: https://issues.apache.org/jira/browse/KAFKA-4682 -Matthias On 6/1/18 2:00 AM, Dinesh Subramanian wrote: > Hi M. Manna, > > Am planning to store

Re: How to gracefully stop Kafka

Thanks guys, I will try this and update to see if that worked. On Fri, Jun 1, 2018 at 1:42 AM, M. Manna wrote: > Regarding graceful shutdown - I have got a response from Jan in the past - > I am simply quoting that below: > > "A gracefully shutdown means the broker is only shutting down when it

Re: Kafka consumer loop exception handling

Than you. In my case i am receiving messages , doing a small transformation and sending to a output topic . If i am running 4 consumers against 4 partitions and one of the consumer dies , will there be duplicate messages sent in this case Since when the new consumer comes up , it will again

Re: Custom AuthenticateCallbackHandler

This feature will be part upcoming Kafka 2.0.0 release. Doc PR is here : https://github.com/apache/kafka/pull/4890 configs here: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L57 On Fri, Jun 1, 2018 at 10:51 PM, Roy van der Valk

Custom AuthenticateCallbackHandler

Dear Kafka community, Can somebody help me setting up a custom AuthenticateCallbackHandler as described in KIP-86 recently added by Rajini Sivaram or point me to good documentation? I described my question in more detail on Stackoverflow:

Re: Frequent consumer rebalances, auto commit failures

1. Any detail logs ? 2. How do you process the records after you polled the records? 3. How much time does it take for every round of poll ? Thanks ! -- Sent from my iPhone On May 28, 2018, at 10:44 PM, Shantanu Deshmukh wrote: Can anyone here help me please? I am at my wit's end. I now have

Re: StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG

You can find the release plan for 2.0 here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820 Beside, as Matthias mentioned you can still work around it as of now by hard-coding the string values in your app: props.put("default.deserialization.exception.handler", ...)

Re: Kafka consumer loop exception handling

This is actually quite nicely explained by Jason Gustafson on this article - https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ It's technically up to the application on how to determine whether message is fully received. If you have database txn

kafka doc question/clarification regarding disk peformance

I'm a newbie reading kafka documentation https://kafka.apache.org/documentation/#persistence says: > the performance of linear writes on a JBOD configuration with six 7200rpm SATA RAID-5 array is ... I am confused. I don't know how the configuration can be both "JBOD" and "RAID-5 array" at the

Re: StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG

Thanks Guozhang !! Any idea when would that be? *Sumit* On 31 May 2018 at 16:48, Guozhang Wang wrote: > Hello Sumit, > > We are going to release 2.0 soon which should contain this fix: > https://issues.apache.org/jira/browse/KAFKA-6825 > > I'm going to cherry-pick it into 1.1 as well so that

Re: Is there expiration for committed Offset in the partition

Hi M. Manna, Am planning to store outside kafka.. will this be a solution ? *Thanks & Regards,* *Dinesh S* On Fri, Jun 1, 2018 at 2:05 PM, M. Manna wrote: > This can happen for two reasons: > > 1) Your offsets are expired and removed. So your consumers don't know where > to start from -

Re: How to gracefully stop Kafka

Regarding graceful shutdown - I have got a response from Jan in the past - I am simply quoting that below: "A gracefully shutdown means the broker is only shutting down when it is not the leader of any partition. Therefore you should not be able to gracefully shut down your entire cluster." That

Re: Is there expiration for committed Offset in the partition

This can happen for two reasons: 1) Your offsets are expired and removed. So your consumers don't know where to start from - earliest means "Start from the beginning" 2) You are actually starting as part of a totally new consumer group - in which case it's as designed too - start from the

Re: Is there expiration for committed Offset in the partition

You should just recommit the same offsets sooner than every 24 hours (or whatever your commit topic retention period is set to). The expiry of offsets is based on the timestamp of the commits. -hans > On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian > wrote: > > Hi, > > Facing duplication in

Is there expiration for committed Offset in the partition

Hi, Facing duplication in below scenario Last commit is happened in 3 days back in the consumer, after that no messages produced in the topic. so no commits.. so after 3 days am stopping and restarting the consumer.. this time i faced duplication issue in the consumer as i have this consumer

Re: Best Practice for Consumer Liveliness and avoid frequent rebalancing

What you are talking about is manual partition assignment, which is different than reassignment upon rebalancing. Consumer informs group coordinator that close() is invoked and that would eventually cause rebalancing. I believe what you are talking about is the rebalance listener. On 1 Jun 2018

Re: How to gracefully stop Kafka

Hi, From my point of view, if you don't have any tool that help you in the management of your broker services, in other to do a rolling restart manually, you should shut down one broker at a time. In this way, you leave time to the broker controller service to balance the active replicas into