Re: ISR shrink rate & IsrExpandsPerSec issue

[Arunkumar]

Make sense, Thank you Mani. I appreciate your time and effort.

ThanksArunkumar Pichaimuthu, PMP 

On Wednesday, June 20, 2018, 1:18:24 AM CDT, Manikumar 
 wrote:  
 
 These metrics are meter type metrics, which tracks  count, mean rate,  1-,
5-, and 15-minute moving averages.
You maybe observing count measure, which gives number of events which have
been marked.
You can try monitoring 1/5/15 miniture averages.

On Wed, Jun 20, 2018 at 12:33 AM Arunkumar 
wrote:

> Hi All,
> I am seeing  IsrShrinksPerSec & IsrExpandsPerSec increases (>0) when one
> or more brokers goes down and comes back up into the cluster. But the
> values should get back to 0 once the servers catches up, is my
> understanding after reading most of the documents. But in our production
> environment (3 node cluster) these values does not reduce and keeps
> increasing. This keeps throwing alerts every few seconds. Any insight is
> highly appreciated.
>
> Thanks in advanceArunkumar Pichaimuthu, PMP
  

ISR shrink rate & IsrExpandsPerSec issue

[Arunkumar]

Hi All,
I am seeing  IsrShrinksPerSec & IsrExpandsPerSec increases (>0) when one or 
more brokers goes down and comes back up into the cluster. But the values 
should get back to 0 once the servers catches up, is my understanding after 
reading most of the documents. But in our production environment (3 node 
cluster) these values does not reduce and keeps increasing. This keeps throwing 
alerts every few seconds. Any insight is highly appreciated.

Thanks in advanceArunkumar Pichaimuthu, PMP

Issue with ISR Shrink and ISR Expand

[Arunkumar]

Hi All
I am facing problem with ISR metrics. We have production cluster of 3 zookeeper 
and 3 broker. We have implemented custom metrics code(Also seeing the same on 
jvisualVM as well). When we initially start the brokers we dis not have any 
issue and it worked fine. But when we restarted a broker, We started seeing ISR 
Shrink and ISR Expand metrics > than 0 and keeps growing and starts throwing 
alerts. Now my question is how do I fix this. Any insight is highly 
appreciated. 
ThanksArunkumar Pichaimuthu, PMP

Re: Monitoring and Alerting

[Arunkumar]

Hi All
Thank you for your time and effort. All have suggested my to look at Prometheus 
etc... I am not looking for a solution. I need metrics on which I can do 
alerting. We are implementing custom alerting mechanism. I am using JMX to get 
all the metrics but as many of you suggested I do not need all of them. I need 
very few to alert and I need details about that metrics on which I can create 
alerts.
Thanks in advance 
ThanksArunkumar Pichaimuthu, PMP 

On Wednesday, April 25, 2018, 11:35:27 PM CDT, Anand, Uttam 
<uttam.an...@bnsf.com> wrote:  
 
 You can use Prometheus for monitoring , AlertManager for alerting and Grafana 
to view the real time graphs by using the metrics scraped by Prometheus.


For gathering the metrics of Kafka and Zookeeper , you can use jmx exporter. 
Also, all metrics will not be useful for humans to read and understand and it 
will be overhead for monitoring apps to process them. So you can whitelist the 
metrics you actually want.


https://www.robustperception.io/monitoring-kafka-with-prometheus


You can add jmx exporter as environment variable in container also.
<https://www.robustperception.io/monitoring-kafka-with-prometheus>

Monitoring Kafka with Prometheus | Robust 
Perception<https://www.robustperception.io/monitoring-kafka-with-prometheus/>
www.robustperception.io
We’ve previously looked at how to monitor Cassandra with Prometheus. Let’s see 
the process for getting metrics from another popular Java application, Kafka. 
Similar to what we did for Cassandra, we download Kafka, the JMX exporter and 
the config file:





From: Will Weber <rwawe...@gmail.com>
Sent: Wednesday, April 25, 2018 11:27:40 PM
To: users@kafka.apache.org
Subject: Re: Monitoring and Alerting

EXTERNAL EMAIL
"Thirding" the use of prometheus, recommend reading this blog post as well:
https://www.robustperception.io/monitoring-kafka-with-prometheus

On Wed, Apr 25, 2018 at 11:39 PM, Jonathan Bethune <
jonathan.beth...@instaclustr.com> wrote:

> I also recommend Prometheus. Works great with JMX for Kafka or any Java
> service.
>
> Datadog is also fine if you want simple and don't spending money. Easy to
> integrate with alerting systems and great visualization.
>
> They have a good blogpost about integrating with Kafka and ZooKeeper:
> https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
>
> On 26 April 2018 at 08:02, Stanislav Antic <stanislav.an...@gmail.com>
> wrote:
>
> > I recommend setting up Prometheus which have Zookeeper exporter (helper
> > program which gives metrics from outside software) and you can use JMX
> > exporter with Kafka.
> >
> > You can find example configs in their repo, which is pretty good and you
> > also have already done Grafana dashboards (https://grafana.com/
> dashboards
> > ):
> > https://github.com/prometheus/jmx_exporter/tree/master/example_configs
> >
> >
> > On Thu, Apr 26, 2018 at 12:05 AM, Arunkumar <pm_arunku...@yahoo.com.
> > invalid>
> > wrote:
> >
> > > HI All
> > > I am working on setting up Monitoring and alerting for our production
> > > cluster. As of now we have a cluster of 3 zookeeper and 3 kafka Brokers
> > > which will expand later.
> > > We are planning for basic metrics (important ones) on which we need to
> > > alert. We are in a process of developing alerting system for our
> > cluster. I
> > > googled and did not get much detail as needed.
> > > I see many say the following ones are important and I understand these
> > are
> > > aggregate metrics across cluster (I may not be
> right).kafka.server:type=
> > > ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=
> > > KafkaController,name=OfflinePartitionsCountkafka.controller:type=
> > > KafkaController,name=ActiveControllerCount
> > > But I am looking for broker,topic etc metrics to alert. Can any one
> > > provide info or point to any docs on this regards will be of great
> > > help.            Thanks in advance
> > >
> > > ThanksArunkumar Pichaimuthu, PMP
> >
> >
> >
> >
> > --
> > Stanislav Antic
> >
>
>
>
> --
>
> *Jonathan Bethune - **Senior Consultant*
>
> JP: +81 70 4069 4357
>
> <https://www.facebook.com/instaclustr>  <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>. This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>  

Monitoring and Alerting

[Arunkumar]

HI All
I am working on setting up Monitoring and alerting for our production cluster. 
As of now we have a cluster of 3 zookeeper and 3 kafka Brokers which will 
expand later.
We are planning for basic metrics (important ones) on which we need to alert. 
We are in a process of developing alerting system for our cluster. I googled 
and did not get much detail as needed.
I see many say the following ones are important and I understand these are 
aggregate metrics across cluster (I may not be 
right).kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=KafkaController,name=OfflinePartitionsCountkafka.controller:type=KafkaController,name=ActiveControllerCount
But I am looking for broker,topic etc metrics to alert. Can any one provide 
info or point to any docs on this regards will be of great help.
Thanks in advance    

ThanksArunkumar Pichaimuthu, PMP

Re: RE: Zookeeper and Kafka JMX metrics

[Arunkumar]

Hi Adrien
Thank you for the response and your time. I had looked at the MBeans exposed by 
zookeeper. I am trying to use this MBeans to port the metrics to UI tools. It 
is pretty straight forward with kafka. But I am not getting any sample code for 
zookeeper. 

ThanksArunkumar Pichaimuthu, PMP 

On Wednesday, February 28, 2018, 3:04:06 PM CST, adrien ruffie 
<adriennolar...@hotmail.fr> wrote:  
 
 #yiv0976467386 #yiv0976467386 -- P 
{margin-top:0;margin-bottom:0;}#yiv0976467386 
Hi Arunkumar,




have you take a look if your MBean are exposed with Zookeeper thank to 
JVisualvm yet ? As like in my screen in attachment.




regards Adrien
De : Arunkumar <pm_arunku...@yahoo.com.INVALID>
Envoyé : mardi 27 février 2018 23:19:33
À : users@kafka.apache.org
Objet : Zookeeper and Kafka JMX metrics Dear Folks
We have plans implementing kafa and zookeeper metrics using java JMX API. We 
were able to successfully implement metrics collection using the MBean exposed 
for kafka. But when we try to do so for zookeeper I do not find much API 
support like we have for kafka. Can someone help if you have any insight.
Thanks in advanceArunkumar Pichaimuthu, PMP
  

Zookeeper and Kafka JMX metrics

[Arunkumar]

Dear Folks
We have plans implementing kafa and zookeeper metrics using java JMX API. We 
were able to successfully implement metrics collection using the MBean exposed 
for kafka. But when we try to do so for zookeeper I do not find much API 
support like we have for kafka. Can someone help if you have any insight.
Thanks in advanceArunkumar Pichaimuthu, PMP

Re: What are reasonable limits for max number of consumer groups per partition and per broker?

[Arunkumar]

Hi Jeff
Number of partition depends on number of consumers on that particular consumer 
group. So you may have to create your partitions based on that.

ThanksArunkumar Pichaimuthu, PMP 

On Monday, November 13, 2017, 5:25:35 PM CST, Jeff Widman 
 wrote:  
 
 We're considering an architecture that would result in 5K-10K consumer
groups consuming from a single topic that has one partition.

What are the reasonable limits for the max number of consumer groups per
partition and per broker?

Can a single broker be the group coordinator for 1K+ consumer groups?

-- 

*Jeff Widman*
jeffwidman.com  | 740-WIDMAN-J (943-6265)
<><
  

Re: Migrating from log4j 1 to log4j 2

[Arunkumar]

Thanks for the info Ismael. Is there any time line for the CR? Please let me 
know.
ThanksArunkumar Pichaimuthu, PMP 

On Tuesday, November 14, 2017, 1:49:55 AM CST, Ismael Juma 
<ism...@juma.me.uk> wrote:  
 
 The broker still has a direct dependency on log4j 1.x. There is a PR in
progress to remove this direct dependency.

Ismael

On Thu, Nov 9, 2017 at 10:17 PM, Arunkumar <pm_arunku...@yahoo.com.invalid>
wrote:

> Hi All
> We have a requirement to migrate log4J 1.x to log4j 2 for our kafka
> brokers using log4j bridge utility. According to Apache Docs the code must
> not call DOMConfigurator or PropertyConfigurator class, But when I dig into
> the code I see on Tools package and other packages they have used
> PropertyConfigurator. I just wanted to double check if any one has any
> insight on this. If above statement is true, it is not feasible to migrate,
> any one please confirm.
> Thanks in advanceArunkumar Pichaimuthu, PMP
  

Re: Kafka Custom Authentication & Authorization

[Arunkumar]

Hi There
We are also trying to do the same and we are trying to over ride 
PlainLoginModule as well. When I add a jar it is not identifying and loading 
the jar. If there is any examples which we can follow will be usefull. Any help 
is highly appreciated.
Thanks in advanceArunkumar Pichaimuthu, PMP 

On Tuesday, November 14, 2017, 5:31:58 AM CST, 陈江枫  
wrote:  
 
 You can implement your own authenticator, check SaslAuthenticator  , and
your own authorizer, check SimpleAuthorizer

2017-11-12 9:39 GMT+08:00 chidigam . :

> Hi All,
> To Authenticate & Authorize the producer and consumers, I want to
> integrate  with third party Entitlement manager. Is there reference
> implementation ?
>
> Any link for doc or comments I appreciate.
>
> Regards
> Bhanu
>
  

Re: Migrating from log4j 1 to log4j 2

[Arunkumar]

Hi Ted
Thanks for quick response. Yes, I know it is used. But my question is if I 
migrate without changing the code will it break anything. We are using 
Confluent Kafka and we cannot modify kafka code. All I can do is change log4j 
1.2.7 jar to latest,  create log4j.xml  and add log4j bridge utility jar. 
Please let me know on this.
ThanksArunkumar Pichaimuthu, PMP 

On Thursday, November 9, 2017, 4:23:43 PM CST, Ted Yu <yuzhih...@gmail.com> 
wrote:  
 
 PropertyConfigurator is used here:

https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/VerifiableLog4jAppender.java#L233

On Thu, Nov 9, 2017 at 2:17 PM, Arunkumar <pm_arunku...@yahoo.com.invalid>
wrote:

> Hi All
> We have a requirement to migrate log4J 1.x to log4j 2 for our kafka
> brokers using log4j bridge utility. According to Apache Docs the code must
> not call DOMConfigurator or PropertyConfigurator class, But when I dig into
> the code I see on Tools package and other packages they have used
> PropertyConfigurator. I just wanted to double check if any one has any
> insight on this. If above statement is true, it is not feasible to migrate,
> any one please confirm.
> Thanks in advanceArunkumar Pichaimuthu, PMP
  

Migrating from log4j 1 to log4j 2

[Arunkumar]

Hi All
We have a requirement to migrate log4J 1.x to log4j 2 for our kafka brokers 
using log4j bridge utility. According to Apache Docs the code must not call 
DOMConfigurator or PropertyConfigurator class, But when I dig into the code I 
see on Tools package and other packages they have used PropertyConfigurator. I 
just wanted to double check if any one has any insight on this. If above 
statement is true, it is not feasible to migrate, any one please confirm.
Thanks in advanceArunkumar Pichaimuthu, PMP

Info on ACL - broker, consumer and producer

[Arunkumar]

Hi All

I am trying to understand how ACL works between broker, producer and consumer. 
I am trying to understand internals of ACL to customize it. I browsed through 
internet and couldn't find much help. Anyone having any documentation or a go 
to resource would be highly appreciated. Thanks in advance.

Thanks
Arunkumar Pichaimuthu, PMP

Consumer group failed with SASL_SSL configuration

[Arunkumar]

Hi All

I configured 3 zookeeper and 3 kafka node cluster with SASL_SSL which is 
working fine. I am able to produce and consume from the command line. But when 
I try to describe consumer group it fails. Please see below for the command and 
the error. Any insight is highly appreciated. 


bin/kafka-consumer-groups --bootstrap-server 
broker1:9094,broker2:9094,broker:9094 --new-consumer --group test-group 
--describe
Note: This will only show information about consumers that use the Java 
consumer API (non-ZooKeeper-based consumers).

Error: Executing consumer group command failed due to Request GROUP_COORDINATOR 
failed on brokers List(broker1:9094 (id: -3 rack: null), broker2:9094 (id: -1 
rack: null), broker3:9094 (id: -2 rack: null))


Thanks
Arunkumar Pichaimuthu, PMP

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

Thank you Vahid

I appreciate you time.

Arunkumar Pichaimuthu, PMP


On Fri, 6/16/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Friday, June 16, 2017, 6:30 PM
 
 Hi Arunkumar,
 
 I'm glad you were able to fix the
 issue. Also glad that the article was 
 helpful.
 
 Regarding Kafka SSL configuration, I'm
 sending some links:
 - Kafka documentation: 
 http://kafka.apache.org/documentation.html#security_ssl
 - Apache Kafka Security 101: 
 
https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
 - Configuring Kafka Security: 
 https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html
 
 I hope they help you get started with
 SSL configuration.
 
 --Vahid
 
 
 
 
 From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
 To:     <users@kafka.apache.org>
 Date:   06/16/2017 03:47 PM
 Subject:        Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 
 
 
 Hi Vahid
 
 I deleted the dataDir and dataLogDir
 and restarted zookeeper,brokers, 
 producers and consumer. Now it
 works
 
 All the messages produced are consumed
 from the producer.
 
 Thanks for all the help. The link you
 shared helped a lot.
 
 I am planning to setup SASL_SSL, I
 appreciate you advice for the same.
 
 Thanks
 Arunkumar Pichaimuthu, PMP
 
 
 On Fri, 6/16/17, Arunkumar <pm_arunku...@yahoo.com.INVALID>
 wrote:
 
  Subject: Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
  To: users@kafka.apache.org
  Date: Friday, June 16, 2017, 4:15 PM
  
  Hi Vahid
  
  I am working on the same use case ):
 .
  As per the document I was trying to
 set ACL's for topic
  which worked and now I am able to
 start my producer without
  error.
  
  Then I set ACL for the consumer and
  when I start my consumer it starts
 without issue. and also
  able to set second ACL for committing
 offsets to group
  arun-group which is also good. 
  
  Current ACLs for resource
  `Group:arun-group`:
         
 User:admin
  has Allow permission for operations:
 All from hosts: *
          User:arun
  has Allow permission for operations:
 All from hosts: *
          User:arun
  has Allow permission for operations:
 Read from hosts: *
         
 User:admin
  has Allow permission for operations:
 Read from hosts: *
  
  Then when I try to get the proper
  listing of offsets in the group I get
 the following error
  
   bin/kafka-consumer-groups
  --bootstrap-server producerhost:9097
 --group arun-group
  --describe --command-config
 etc/kafka/producer.properties
  
  Note: This will only show information
  about consumers that use the Java
 consumer API
  (non-ZooKeeper-based consumers).
  
  [2017-06-16 16:05:42,535] INFO
  Successfully logged in.
 
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
  Error: Executing consumer group
 command
  failed due to The group coordinator is
 not available.
  
  Also I don't see the message producer
  posted on consumer.
  
  
  I am using the confluent opensource
  kafka which is bundled with 
  Zookeeper - 3.4.6
  Kafka - 10.2.0
  
  Please let me know if you need any
 more
  info. I appreciate your time.
  
  Thanks
  Arunkumar Pichaimuthu, PMP
  
 
 
  On Fri, 6/16/17, Vahid S Hashemian
  <vahidhashem...@us.ibm.com>
  wrote:
  
   Subject: Re:
  UNKNOWN_TOPIC_OR_PARTITION with
 SASL_PLAINTEXT ACL
   To: users@kafka.apache.org
   Date: Friday, June 16, 2017,
 1:56 PM
  
   Hi Arunkumar,
  
   Were you trying the same steps
 in the
   document when you got this
 error? Or 
   you are working on a different
 use
   case?
   Also, I might have missed it in
   previous emails. What version of
 Kafka
  are 
   you using?
  
   Thanks.
   --Vahid
  
  
  
   From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
   To:     <users@kafka.apache.org>
   Date:   06/16/2017 10:22
 AM
   Subject:     
   Re:
   UNKNOWN_TOPIC_OR_PARTITION with
  SASL_PLAINTEXT ACL
  
  
  
    Hi Vahid
  
   Thank you for sharing link to
 set it
   up. It is really a very useful 
   document. When I ran describe
 command
   for group I see this error
  
    bin/kafka-consumer-groups
   --bootstrap-server host:9097
  --describe --group 
   arun-group --command-config
   etc/kafka/producer.properties
   Note: This will only show
 information
   about consumers that use the
 Java 
   consumer API
 (non-ZooKeeper-based
   consumers).
  
   [2017-06-16 11:32:23,790] INFO
   Successfully logged in. 
  
 
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
   Error: Executing consumer group
  command
   failed due to The group 
   coordinator is not available.
  
  
   I googled to figure out the
 issue and
   many say that it may be because
 of 
   the port which I am not
 convinced.
  Any
 

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

Hi Vahid

I deleted the dataDir and dataLogDir and restarted zookeeper,brokers, producers 
and consumer. Now it works

All the messages produced are consumed from the producer.

Thanks for all the help. The link you shared helped a lot.

I am planning to setup SASL_SSL, I appreciate you advice for the same.

Thanks
Arunkumar Pichaimuthu, PMP


On Fri, 6/16/17, Arunkumar <pm_arunku...@yahoo.com.INVALID> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Friday, June 16, 2017, 4:15 PM
 
 Hi Vahid
 
 I am working on the same use case ): .
 As per the document I was trying to set ACL's for topic
 which worked and now I am able to start my producer without
 error.
 
 Then I set ACL for the consumer and
 when I start my consumer it starts without issue. and also
 able to set second ACL for committing offsets to group
 arun-group which is also good. 
 
 Current ACLs for resource
 `Group:arun-group`:
         User:admin
 has Allow permission for operations: All from hosts: *
         User:arun
 has Allow permission for operations: All from hosts: *
         User:arun
 has Allow permission for operations: Read from hosts: *
         User:admin
 has Allow permission for operations: Read from hosts: *
 
 Then when I try to get the proper
 listing of offsets in the group I get the following error
 
  bin/kafka-consumer-groups
 --bootstrap-server producerhost:9097 --group arun-group
 --describe --command-config etc/kafka/producer.properties
 
 Note: This will only show information
 about consumers that use the Java consumer API
 (non-ZooKeeper-based consumers).
 
 [2017-06-16 16:05:42,535] INFO
 Successfully logged in.
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
 Error: Executing consumer group command
 failed due to The group coordinator is not available.
 
 Also I don't see the message producer
 posted on consumer.
 
 
 I am using the confluent opensource
 kafka which is bundled with 
 Zookeeper - 3.4.6
 Kafka - 10.2.0
 
 Please let me know if you need any more
 info. I appreciate your time.
 
 Thanks
 Arunkumar Pichaimuthu, PMP
 
 
 On Fri, 6/16/17, Vahid S Hashemian
 <vahidhashem...@us.ibm.com>
 wrote:
 
  Subject: Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
  To: users@kafka.apache.org
  Date: Friday, June 16, 2017, 1:56 PM
  
  Hi Arunkumar,
  
  Were you trying the same steps in the
  document when you got this error? Or 
  you are working on a different use
  case?
  Also, I might have missed it in
  previous emails. What version of Kafka
 are 
  you using?
  
  Thanks.
  --Vahid
  
  
  
  From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
  To:     <users@kafka.apache.org>
  Date:   06/16/2017 10:22 AM
  Subject:        Re:
  UNKNOWN_TOPIC_OR_PARTITION with
 SASL_PLAINTEXT ACL
  
  
  
   Hi Vahid
  
  Thank you for sharing link to set it
  up. It is really a very useful 
  document. When I ran describe command
  for group I see this error
  
   bin/kafka-consumer-groups
  --bootstrap-server host:9097
 --describe --group 
  arun-group --command-config
  etc/kafka/producer.properties
  Note: This will only show information
  about consumers that use the Java 
  consumer API (non-ZooKeeper-based
  consumers).
  
  [2017-06-16 11:32:23,790] INFO
  Successfully logged in. 
 
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
  Error: Executing consumer group
 command
  failed due to The group 
  coordinator is not available.
  
  
  I googled to figure out the issue and
  many say that it may be because of 
  the port which I am not convinced.
 Any
  help is highly appreciated.
  
  Thanks
  Arunkumar Pichaimuthu, PMP
  
 
 
  On Thu, 6/15/17, Vahid S Hashemian
  <vahidhashem...@us.ibm.com>
  wrote:
  
   Subject: Re:
  UNKNOWN_TOPIC_OR_PARTITION with
 SASL_PLAINTEXT ACL
   To: users@kafka.apache.org
   Date: Thursday, June 15, 2017,
 6:49
  PM
   
   Hi Arunkumar,
   
   Could you please take a look at
 this
  article:
   
   https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
   The error message you posted
 earlier
  suggests
   there is some missing ACL 
   (as indicated in
   the article).
   
   Let me know
   if that doesn't resolve the
 issue.
  Thanks.
   --Vahid
   
   
   
   
   From:  
   Arunkumar <pm_arunku...@yahoo.com.INVALID>
   To:     <users@kafka.apache.org>
   Date:   06/15/2017 04:37
 PM
   Subject:       
  Re:
   UNKNOWN_TOPIC_OR_PARTITION with
  SASL_PLAINTEXT ACL
   
   
   
   Hi Vahid
   
   Thank you for quick
   response.
   
   I set the ACL for
   topic and also created jaas
 permission
  as per the 
   document for both producer and
  consumer. I have
   set what I have posted 
   below. Do I need to
   set ACL like we set for Topics
  --  bin/kafka-acls 
   --topic * --add -allow-host

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

Hi Vahid

I am working on the same use case ): . As per the document I was trying to set 
ACL's for topic which worked and now I am able to start my producer without 
error.

Then I set ACL for the consumer and when I start my consumer it starts without 
issue. and also able to set second ACL for committing offsets to group 
arun-group which is also good. 

Current ACLs for resource `Group:arun-group`:
User:admin has Allow permission for operations: All from hosts: *
User:arun has Allow permission for operations: All from hosts: *
User:arun has Allow permission for operations: Read from hosts: *
User:admin has Allow permission for operations: Read from hosts: *

Then when I try to get the proper listing of offsets in the group I get the 
following error

 bin/kafka-consumer-groups --bootstrap-server producerhost:9097 --group 
arun-group --describe --command-config etc/kafka/producer.properties

Note: This will only show information about consumers that use the Java 
consumer API (non-ZooKeeper-based consumers).

[2017-06-16 16:05:42,535] INFO Successfully logged in. 
(org.apache.kafka.common.security.authenticator.AbstractLogin)
Error: Executing consumer group command failed due to The group coordinator is 
not available.

Also I don't see the message producer posted on consumer.


I am using the confluent opensource kafka which is bundled with 
Zookeeper - 3.4.6
Kafka - 10.2.0

Please let me know if you need any more info. I appreciate your time.

Thanks
Arunkumar Pichaimuthu, PMP


On Fri, 6/16/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Friday, June 16, 2017, 1:56 PM
 
 Hi Arunkumar,
 
 Were you trying the same steps in the
 document when you got this error? Or 
 you are working on a different use
 case?
 Also, I might have missed it in
 previous emails. What version of Kafka are 
 you using?
 
 Thanks.
 --Vahid
 
 
 
 From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
 To:     <users@kafka.apache.org>
 Date:   06/16/2017 10:22 AM
 Subject:        Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 
 
 
  Hi Vahid
 
 Thank you for sharing link to set it
 up. It is really a very useful 
 document. When I ran describe command
 for group I see this error
 
  bin/kafka-consumer-groups
 --bootstrap-server host:9097 --describe --group 
 arun-group --command-config
 etc/kafka/producer.properties
 Note: This will only show information
 about consumers that use the Java 
 consumer API (non-ZooKeeper-based
 consumers).
 
 [2017-06-16 11:32:23,790] INFO
 Successfully logged in. 
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
 Error: Executing consumer group command
 failed due to The group 
 coordinator is not available.
 
 
 I googled to figure out the issue and
 many say that it may be because of 
 the port which I am not convinced. Any
 help is highly appreciated.
 
 Thanks
 Arunkumar Pichaimuthu, PMP
 
 
 On Thu, 6/15/17, Vahid S Hashemian
 <vahidhashem...@us.ibm.com>
 wrote:
 
  Subject: Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
  To: users@kafka.apache.org
  Date: Thursday, June 15, 2017, 6:49
 PM
  
  Hi Arunkumar,
  
  Could you please take a look at this
 article:
  
  https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
  The error message you posted earlier
 suggests
  there is some missing ACL 
  (as indicated in
  the article).
  
  Let me know
  if that doesn't resolve the issue.
 Thanks.
  --Vahid
  
  
  
  
  From:  
  Arunkumar <pm_arunku...@yahoo.com.INVALID>
  To:     <users@kafka.apache.org>
  Date:   06/15/2017 04:37 PM
  Subject:       
 Re:
  UNKNOWN_TOPIC_OR_PARTITION with
 SASL_PLAINTEXT ACL
  
  
  
  Hi Vahid
  
  Thank you for quick
  response.
  
  I set the ACL for
  topic and also created jaas permission
 as per the 
  document for both producer and
 consumer. I have
  set what I have posted 
  below. Do I need to
  set ACL like we set for Topics
 --  bin/kafka-acls 
  --topic * --add -allow-host host:9097
  --allow-principal User:arun 
  --operation
  Write --authorizer-properties
 zookeeper.connect=host:2182 ?
  
  Please let me know. If you need all
  configuration for zookeeper, Broker, 
  producer and consumer. I can share it
 as well.
  Thanks in advance
  
  
  KafkaServer {
    
 
 org.apache.kafka.common.security.plain.PlainLoginModule
  required
     username="admin"
     password="admin-secret"
    
 user_admin="admin-secret"
     user_arun="Arun123";
  };
  
  Client {
         
 
 org.apache.kafka.common.security.plain.PlainLoginModule
  required
         
  username="arun"
         
  password="Arun123";
  };
  
  KafkaClient {
         
 
 org.apache.kafka.common.security.plain.PlainLog

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

 Hi Vahid

Thank you for sharing link to set it up. It is really a very useful document. 
When I ran describe command for group I see this error

 bin/kafka-consumer-groups --bootstrap-server host:9097 --describe --group 
arun-group --command-config etc/kafka/producer.properties
Note: This will only show information about consumers that use the Java 
consumer API (non-ZooKeeper-based consumers).

[2017-06-16 11:32:23,790] INFO Successfully logged in. 
(org.apache.kafka.common.security.authenticator.AbstractLogin)
Error: Executing consumer group command failed due to The group coordinator is 
not available.


I googled to figure out the issue and many say that it may be because of the 
port which I am not convinced. Any help is highly appreciated.

Thanks
Arunkumar Pichaimuthu, PMP


On Thu, 6/15/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Thursday, June 15, 2017, 6:49 PM
 
 Hi Arunkumar,
 
 Could you please take a look at this article:
 
 https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
 The error message you posted earlier suggests
 there is some missing ACL 
 (as indicated in
 the article).
 
 Let me know
 if that doesn't resolve the issue. Thanks.
 --Vahid
 
 
 
 
 From:  
 Arunkumar <pm_arunku...@yahoo.com.INVALID>
 To:     <users@kafka.apache.org>
 Date:   06/15/2017 04:37 PM
 Subject:        Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 
 
 
 Hi Vahid
 
 Thank you for quick
 response.
 
 I set the ACL for
 topic and also created jaas permission as per the 
 document for both producer and consumer. I have
 set what I have posted 
 below. Do I need to
 set ACL like we set for Topics --  bin/kafka-acls 
 --topic * --add -allow-host host:9097
 --allow-principal User:arun 
 --operation
 Write --authorizer-properties zookeeper.connect=host:2182 ?
 
 Please let me know. If you need all
 configuration for zookeeper, Broker, 
 producer and consumer. I can share it as well.
 Thanks in advance
 
 
 KafkaServer {
   
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_arun="Arun123";
 };
 
 Client {
        
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
        
 username="arun"
        
 password="Arun123";
 };
 
 KafkaClient {
        
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
        
 username="arun"
        
 password="Arun123";
 };
 
 Thanks
 Arunkumar Pichaimuthu, PMP
 
 
 On Thu, 6/15/17, Vahid S Hashemian <vahidhashem...@us.ibm.com>
 wrote:
 
  Subject: Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
  To: users@kafka.apache.org
  Date: Thursday, June 15, 2017, 6:16 PM
  
  Hi Arunkumar,
  
  Have you given your Kafka
 consumer/producer
  necessary permissions to
 
  consume/produce
 
 messages?
  
  --Vahid
  
  
  
  From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
  To:     <users@kafka.apache.org>
  Date:   06/15/2017 04:07 PM
 
 Subject:        UNKNOWN_TOPIC_OR_PARTITION
  with SASL_PLAINTEXT ACL
  
  
  
  Hi 
  
  I am setting up ACL with
 SALS_PLAINTEXT. My
  zookeeper and broker
 starts 
  without error.
  But
 when I try to start my consumer or if I send message 
  through a producer it throws an exception
 (Both
  producer and consumer are 
  kafka CLI)
  Stack trace for my
 consumer below. Any insight
  is highly
 appreciated. 
  Thanks in advance
  
 
 bin/kafka-console-consumer
  --topic sample1
 --from-beginning 
 
 --consumer.config=etc/kafka/consumer.properties 
  --bootstrap-server 
 
 hostname:9097
  [2017-06-15 17:21:45,286]
 INFO ConsumerConfig
  values:
          auto.commit.interval.ms
  = 5000
         
 auto.offset.reset =
  earliest
          bootstrap.servers =
  [hostname:9097]
         
 check.crcs =
  true
      
    client.id =
         
 connections.max.idle.ms = 54
        
  enable.auto.commit = true
         
 exclude.internal.topics = true
         
 fetch.max.bytes = 52428800
         
 fetch.max.wait.ms = 500
         
 fetch.min.bytes = 1
          group.id =
 test-consumer-group
         
 heartbeat.interval.ms = 1000
         
 interceptor.classes = null
         
 key.deserializer = class 
 
 org.apache.kafka.common.serialization.ByteArrayDeserializer
          max.partition.fetch.bytes =
  1048576
         
 max.poll.interval.ms =
  30
          max.poll.records = 500
          metadata.max.age.ms = 30
          metric.reporters = []
          metrics.num.samples = 2
          metrics.recording.level = INFO
          metrics.sample.window.ms =
 3
         
 partition.assignment.strategy =
  [class 
 
 org.apache.kafka.clients.consumer.Ra

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

I also created ACL for both producer and consumer. Still no luck


bin/kafka-acls --producer host:9097 --topic sample1 --add -allow-host 
hostname9097 --allow-principal User:arun --authorizer-properties 
zookeeper.connect=zookeeperhost:2182
 
 
 bin/kafka-acls --consumer host:9097 --topic sample1 --group 
test-consumer-group --add -allow-host hostname:9097 --allow-principal User:arun 
--authorizer-properties zookeeper.connect=zookeeperhost:2182

Thanks
Arunkumar Pichaimuthu, PMP


On Thu, 6/15/17, Arunkumar <pm_arunku...@yahoo.com.INVALID> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Thursday, June 15, 2017, 6:33 PM
 
 Hi Vahid
 
 Thank you for quick response.
 
 I set the ACL for topic and also
 created jaas permission as per the document for both
 producer and consumer. I have set what I have posted below.
 Do I need to set ACL like we set for Topics -- 
 bin/kafka-acls --topic * --add -allow-host host:9097
 --allow-principal User:arun --operation Write
 --authorizer-properties zookeeper.connect=host:2182 ? Please
 let me know. If you need all configuration for zookeeper,
 Broker, producer and consumer. I can share it as well.
 Thanks in advance
 
 
 KafkaServer {
   
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_arun="Arun123";
 };
 
 Client {
        
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
        
 username="arun"
        
 password="Arun123";
 };
 
 KafkaClient {
        
 org.apache.kafka.common.security.plain.PlainLoginModule
 required
        
 username="arun"
        
 password="Arun123";
 };
 
 Thanks
 Arunkumar Pichaimuthu, PMP
 
 
 On Thu, 6/15/17, Vahid S Hashemian
 <vahidhashem...@us.ibm.com>
 wrote:
 
  Subject: Re:
 UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
  To: users@kafka.apache.org
  Date: Thursday, June 15, 2017, 6:16
 PM
  
  Hi Arunkumar,
  
  Have you given your Kafka
 consumer/producer
  necessary permissions to 
  consume/produce
  messages?
  
  --Vahid
  
  
  
  From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
  To:     <users@kafka.apache.org>
  Date:   06/15/2017 04:07 PM
  Subject:       
 UNKNOWN_TOPIC_OR_PARTITION
  with SASL_PLAINTEXT ACL
  
  
  
  Hi 
  
  I am setting up ACL with
 SALS_PLAINTEXT. My
  zookeeper and broker starts 
  without error.
  But when I try to start my consumer or
 if I send message 
  through a producer it throws an
 exception (Both
  producer and consumer are 
  kafka CLI)
  Stack trace for my consumer below. Any
 insight
  is highly appreciated. 
  Thanks in advance
  
  bin/kafka-console-consumer
  --topic sample1 --from-beginning 
 
 --consumer.config=etc/kafka/consumer.properties 
  --bootstrap-server 
  hostname:9097
  [2017-06-15 17:21:45,286] INFO
 ConsumerConfig
  values:
          auto.commit.interval.ms
  = 5000
          auto.offset.reset =
  earliest
          bootstrap.servers =
  [hostname:9097]
          check.crcs =
  true
          client.id =
          connections.max.idle.ms =
 54
          enable.auto.commit = true
          exclude.internal.topics =
 true
          fetch.max.bytes =
 52428800
          fetch.max.wait.ms = 500
          fetch.min.bytes = 1
          group.id =
 test-consumer-group
          heartbeat.interval.ms =
 1000
          interceptor.classes =
 null
          key.deserializer = class 
 
 org.apache.kafka.common.serialization.ByteArrayDeserializer
          max.partition.fetch.bytes
 =
  1048576
          max.poll.interval.ms =
  30
          max.poll.records = 500
          metadata.max.age.ms =
 30
          metric.reporters = []
          metrics.num.samples = 2
          metrics.recording.level =
 INFO
          metrics.sample.window.ms =
 3
         
 partition.assignment.strategy =
  [class 
 
 org.apache.kafka.clients.consumer.RangeAssignor]
          receive.buffer.bytes =
 65536
          reconnect.backoff.ms = 50
          request.timeout.ms =
 305000
          retry.backoff.ms = 100
          sasl.jaas.config = null
          sasl.kerberos.kinit.cmd =
  /usr/bin/kinit
         
  sasl.kerberos.min.time.before.relogin
 = 6
          sasl.kerberos.service.name
 =
  null
         
  sasl.kerberos.ticket.renew.jitter =
 0.05
   
       
 sasl.kerberos.ticket.renew.window.factor = 0.8
          sasl.mechanism = PLAIN
          security.protocol =
  SASL_PLAINTEXT
          send.buffer.bytes
  = 131072
          session.timeout.ms =
  1
          ssl.cipher.suites =
  null
          ssl.enabled.protocols =
  [TLSv1.2, TLSv1.1, TLSv1]
         
  ssl.endpoint.identification.algorithm
 = null
          ssl.key.password = null
          ssl.keymanager.

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

Hi Vahid

Thank you for quick response.

I set the ACL for topic and also created jaas permission as per the document 
for both producer and consumer. I have set what I have posted below. Do I need 
to set ACL like we set for Topics --  bin/kafka-acls --topic * --add 
-allow-host host:9097 --allow-principal User:arun --operation Write 
--authorizer-properties zookeeper.connect=host:2182 ? Please let me know. If 
you need all configuration for zookeeper, Broker, producer and consumer. I can 
share it as well. Thanks in advance


KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
   user_arun="Arun123";
};

Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="arun"
password="Arun123";
};

KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
    username="arun"
password="Arun123";
};

Thanks
Arunkumar Pichaimuthu, PMP


On Thu, 6/15/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote:

 Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
 To: users@kafka.apache.org
 Date: Thursday, June 15, 2017, 6:16 PM
 
 Hi Arunkumar,
 
 Have you given your Kafka consumer/producer
 necessary permissions to 
 consume/produce
 messages?
 
 --Vahid
 
 
 
 From:   Arunkumar <pm_arunku...@yahoo.com.INVALID>
 To:     <users@kafka.apache.org>
 Date:   06/15/2017 04:07 PM
 Subject:        UNKNOWN_TOPIC_OR_PARTITION
 with SASL_PLAINTEXT ACL
 
 
 
 Hi 
 
 I am setting up ACL with SALS_PLAINTEXT. My
 zookeeper and broker starts 
 without error.
 But when I try to start my consumer or if I send message 
 through a producer it throws an exception (Both
 producer and consumer are 
 kafka CLI)
 Stack trace for my consumer below. Any insight
 is highly appreciated. 
 Thanks in advance
 
 bin/kafka-console-consumer
 --topic sample1 --from-beginning 
 --consumer.config=etc/kafka/consumer.properties 
 --bootstrap-server 
 hostname:9097
 [2017-06-15 17:21:45,286] INFO ConsumerConfig
 values:
         auto.commit.interval.ms
 = 5000
         auto.offset.reset =
 earliest
         bootstrap.servers =
 [hostname:9097]
         check.crcs =
 true
         client.id =
         connections.max.idle.ms = 54
         enable.auto.commit = true
         exclude.internal.topics = true
         fetch.max.bytes = 52428800
         fetch.max.wait.ms = 500
         fetch.min.bytes = 1
         group.id = test-consumer-group
         heartbeat.interval.ms = 1000
         interceptor.classes = null
         key.deserializer = class 
 org.apache.kafka.common.serialization.ByteArrayDeserializer
         max.partition.fetch.bytes =
 1048576
         max.poll.interval.ms =
 30
         max.poll.records = 500
         metadata.max.age.ms = 30
         metric.reporters = []
         metrics.num.samples = 2
         metrics.recording.level = INFO
         metrics.sample.window.ms = 3
         partition.assignment.strategy =
 [class 
 org.apache.kafka.clients.consumer.RangeAssignor]
         receive.buffer.bytes = 65536
         reconnect.backoff.ms = 50
         request.timeout.ms = 305000
         retry.backoff.ms = 100
         sasl.jaas.config = null
         sasl.kerberos.kinit.cmd =
 /usr/bin/kinit
        
 sasl.kerberos.min.time.before.relogin = 6
         sasl.kerberos.service.name =
 null
        
 sasl.kerberos.ticket.renew.jitter = 0.05
  
       sasl.kerberos.ticket.renew.window.factor = 0.8
         sasl.mechanism = PLAIN
         security.protocol =
 SASL_PLAINTEXT
         send.buffer.bytes
 = 131072
         session.timeout.ms =
 1
         ssl.cipher.suites =
 null
         ssl.enabled.protocols =
 [TLSv1.2, TLSv1.1, TLSv1]
        
 ssl.endpoint.identification.algorithm = null
         ssl.key.password = null
         ssl.keymanager.algorithm =
 SunX509
         ssl.keystore.location =
 null
         ssl.keystore.password =
 null
         ssl.keystore.type = JKS
         ssl.protocol = TLS
         ssl.provider = null
         ssl.secure.random.implementation =
 null
         ssl.trustmanager.algorithm
 = PKIX
         ssl.truststore.location =
 path.truststore
        
 ssl.truststore.password = [hidden]
      
   ssl.truststore.type = JKS
        
 value.deserializer = class 
 org.apache.kafka.common.serialization.ByteArrayDeserializer
 
 (org.apache.kafka.clients.consumer.ConsumerConfig)
 [2017-06-15 17:21:45,438] INFO Successfully
 logged in. 
 (org.apache.kafka.common.security.authenticator.AbstractLogin)
 [2017-06-15 17:21:45,522] INFO Kafka version :
 0.10.2.1-cp1 
 (org.apache.kafka.common.utils.AppInfoParser)
 [2017-06-15 17:21:45,523] INFO Kafka commitId :
 078e7dc02a100018 
 (org.apache.kafka.common.utils.AppInfoParser)
 [2017-06-15 17:21:45,781] 

UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

[Arunkumar]

 while fetching metadata with correlation 
id 13 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,013] WARN Error while fetching metadata with correlation 
id 14 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,117] WARN Error while fetching metadata with correlation 
id 15 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,220] WARN Error while fetching metadata with correlation 
id 16 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,323] WARN Error while fetching metadata with correlation 
id 17 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,426] WARN Error while fetching metadata with correlation 
id 18 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,530] WARN Error while fetching metadata with correlation 
id 19 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,634] WARN Error while fetching metadata with correlation 
id 20 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)


Thanks
Arunkumar Pichaimuthu, PMP

Login using LAP for Kafka

[Arunkumar]

Hi 

I have a requirement to configure LDAP Login for Kafka. I see there is a 
discussion going on to integrate it with future release of Kafka version. Is 
there any one who have done it ( I am sure some one would have done it ):- ) 
can help with sample configuration with code. I see we can extend the 
PlainLoginModule class and write our own implementation any insight is highly 
appreciated. Thanks in advance.

Thanks
Arunkumar Pichaimuthu, PMP

LDAP integration with kafka brokers

[Arunkumar]

Hi Group,

We have a requirement to integrate Kafka cluster with LDAP for authentication. 
I googled and I did not find much about it.
Out requirement is when a producer sends a message to a broker it would send 
producer credentials like topic to which it wants to write to and userId with 
passcode. We would like to authenticate it against out Enterprise LDAP server 
for authenticity. If there is no LDAP support available,  we are planning to 
customize the code. Any insight on this is highly appreciated.

Thanks in advance 
Arunkumar Pichaimuthu, PMP

Subscribe to mailing list

[Arunkumar]

Hi There
I would like to subscribe to this mailing list and know more about kafka. 
Please add me to the list. Thanks in advance

Thanks
Arunkumar Pichaimuthu, PMP

RE: JSON parsing causing rebalance to fail

[Arunkumar Srambikkal (asrambik)]

Update : 

Turns out this error happens in 2 scenarios 

1. When there is a mis-match between the broker and zookeeper libs inside of 
your process (found that from stackoverflow)

2.Apparetly when anything that uses scala parser combinators libs (in our case 
scala.util.parsing.json.JSON) runs within the same process as your consumer

1 is easily fixed. 2 happens consistently and it is often caused by other libs 
that you use, which makes it impossible to use kafka at all.

It's quite silly but totally disabling. 

We would appreciate it a lot, if someone could provide us a workaround  / point 
us to the exact issue so that we can custom patch it / provide a patch 
ourselves. 

Thanks
Arun

From: Arunkumar Srambikkal (asrambik) 
Sent: Wednesday, March 04, 2015 5:27 PM
To: users@kafka.apache.org
Subject: JSON parsing causing rebalance to fail

Hi, 

When I start a new consumer, it throws a Rebalance exception. 

However I hit it only on some machines where the run time libraries are 
different

The stack given below is what I encounter - is this a known issue? 

I saw this Jira but it's not resolved  so thought to confirm - 
https://issues.apache.org/jira/browse/KAFKA-1405

Thanks
Arun

[2015-03-04 14:30:37,609] INFO [name], exception during rebalance  
(kafka.consumer.ZookeeperConsumerConnector)

kafka.common.KafkaException: Failed to parse the broker info from zookeeper: 
{jmx_port:-1,timestamp:1425459616502,host:1.1.1.1,version:1,port:64356}
    at kafka.cluster.Broker$.createBroker(Broker.scala:35)
    ..
Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to 
java.lang.Integer
    at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:105)
    at kafka.cluster.Broker$.createBroker(Broker.scala:40)

RE: Kafka producer failed to send but actually does

[Arunkumar Srambikkal (asrambik)]

Thanks for responding. 

I was creating an instance of kafka.server.KafkaServer in my code for running 
some tests and this was what I referred to by an embedded broker. 

The scenario you described was what was happening. In my case when I kill my 
broker, it fails to send an ack. I added handling of duplicates and resolved 
this issue.

Thanks
Arun

-Original Message-
From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID] 
Sent: Tuesday, March 03, 2015 11:13 PM
To: users@kafka.apache.org
Subject: Re: Kafka producer failed to send but actually does

What do you mean by Kafka embedded broker?
Anyway, this could happen. For example, producer sends message to broker.
After that some network issue occurs and the producer did not got confirmation 
from broker, so the producer thought the send failed. But the broker actually 
got the message. The produce is expected to resend the message, so broker will 
have duplicate messages, and that¹s also why we say Kafka guarantees at least 
once.

-Jiangjie (Becket) Qin

On 3/3/15, 4:01 AM, Arunkumar Srambikkal (asrambik) asram...@cisco.com
wrote:

Hi,

I'm running some tests with the Kafka embedded broker and I see cases 
where the producer gets the FailedToSendMessageException but in reality 
the message is transferred and consumer gets it

Is this expected / known issue?

Thanks
Arun

My producer config =

props.put(producer.type, sync)
props.put(serializer.class,  kafka.serializer.StringEncoder);
props.put(partitioner.class, com.test.PartMe);
props.put(metadata.broker.list,  127.0.0.1:+port);
props.put(request.required.acks, -1);
props.put(message.send.max.retries, 0)

JSON parsing causing rebalance to fail

[Arunkumar Srambikkal (asrambik)]

Hi,

When I start a new consumer, it throws a Rebalance exception.

However I hit it only on some machines where the run time libraries are 
different

The stack given below is what I encounter - is this a known issue?

I saw this Jira but it's not resolved  so thought to confirm - 
https://issues.apache.org/jira/browse/KAFKA-1405

Thanks
Arun

[2015-03-04 14:30:37,609] INFO [name], exception during rebalance  
(kafka.consumer.ZookeeperConsumerConnector)

kafka.common.KafkaException: Failed to parse the broker info from zookeeper: 
{jmx_port:-1,timestamp:1425459616502,host:1.1.1.1,version:1,port:64356}
at kafka.cluster.Broker$.createBroker(Broker.scala:35)

Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to 
java.lang.Integer
at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:105)
at kafka.cluster.Broker$.createBroker(Broker.scala:40)

Kafka producer failed to send but actually does

[Arunkumar Srambikkal (asrambik)]

Hi, 

I'm running some tests with the Kafka embedded broker and I see cases where the 
producer gets the FailedToSendMessageException but in reality the message is 
transferred and consumer gets it 

Is this expected / known issue? 

Thanks
Arun

My producer config = 

props.put(producer.type, sync)
props.put(serializer.class,  kafka.serializer.StringEncoder);
props.put(partitioner.class, com.test.PartMe);
props.put(metadata.broker.list,  127.0.0.1:+port); 
props.put(request.required.acks, -1);   
props.put(message.send.max.retries, 0)

RE: Simple Consumer and offsets

[Arunkumar Srambikkal (asrambik)]

If I may use the same thread to discuss the exact same issue  

Assuming one can store the offset in an external location (redis/db etc), along 
with the rest of the state that a program requires, wouldn't it be possible to 
manage things such that, you use the High Level API with auto commit turned off 
and do your custom offset management followed by the kafka commit api call 
(probably delayed to give a breather to zookeeper)? 

That way in the failure scenario, the high level consumer offset would ALWAYS 
be only smaller than what is actually valid and you can skip forward and avoid 
using the simple consumer.

I assume one needs the simple consumer in the offset management use case, only 
we want to skip back to an older offset / use Kafka for storing offsets? 

I was trying to handle the customer failure scenario but avoiding the simple 
consumer and all the complexities it ensues. 

Does this work or is there anything wrong with this picture? 

Thanks
Arun

On Thu, Feb 19, 2015 at 03:29:19PM +, Suren wrote:
 We are using the High Level Consumer API to interact with Kafka for our 
 normal use cases.
 
 However, on consumer restart in the case of consumer failures, we want 
 to be able to manually reset offsets in certain situations.
 And ideally we'd like to use the same api in 0.8.1 and 0.8.2. :-) It 
 looked like instantiating a SimpleConsumer just to reset offsets on restart 
 was a viable option, while continuing to use the High Level Consumer for our 
 normal operations. Not sure if there is a better way that is compatible 
 across 0.8.1 and 0.8.2.
 -Suren
  
 
  On Thursday, February 19, 2015 10:25 AM, Joel Koshy 
 jjkosh...@gmail.com wrote:

 
  Not sure what you mean by using the SimpleConsumer on failure 
 recovery. Can you elaborate on this?
 
 On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote:
  Haven't used either one now. Sounds like 0.8.2.1 will help.
  We are using the High Level Consumer generally but are thinking to use the 
  SimpleConsumer on failure recovery to set the offsets.
  Is that the recommended approach for this use case?
  Thanks.
  -Suren
   
  
   On Thursday, February 19, 2015 9:40 AM, Joel Koshy 
  jjkosh...@gmail.com wrote:
 
  
   Are you using it from Java or Scala? i.e., are you using the  
 javaapi.SimpleConsumer or kafka.consumer.SimpleConsumer
  
  In 0.8.2 javaapi we explicitly set version 0 of the 
  OffsetCommitRequest/OffsetFetchRequest which means it will 
  commit/fetch to/from ZooKeeper only. If you use the scala API you 
  can create an OffsetCommitRequest with version set to 1 (which will 
  allow you to commit to Kafka).
  
  Since we are doing an 0.8.2.1 release we will make the above more 
  consistent. i.e., you can create OffsetCommitRequests with version 1 
  even from the javaapi. I will be updating the documentation on this 
  to make it clearer.
  
  Thanks,
  
  Joel
  
  On Thu, Feb 19, 2015 at 02:28:32PM +, Suren wrote:
   Joel,
   Looking at SimpleConsumer in the 0.8.2 code, it is using 
   OffsetCommitRequest and sending that over to a broker.
   Is the broker storing that in ZK?
   -Suren

   
On Tuesday, February 17, 2015 12:22 PM, Joel Koshy 
   jjkosh...@gmail.com wrote:
  
   
Hi Chris,
   
   In 0.8.2, the simple consumer Java API supports 
   committing/fetching offsets that are stored in ZooKeeper. You 
   don't need to issue any ConsumerMetadataRequest for this. 
   Unfortunately, the API currently does not support fetching offsets that 
   are stored in Kafka.
   
   Thanks,
   
   Joel
   
   On Mon, Feb 16, 2015 at 05:02:08PM -0500, Christopher Piggott wrote:
Hi,

I am still using 0.8.1.1 because of the CPU use concerns.

I'm confused about why the SimpleConsumer has:

OffsetCommitResponse commitOffsets(OffsetCommitRequest request)

and

OffsetFetchResponse fetchOffsets(OffsetFetchRequest request)

but no way that I can see to issue a ConsumerMetadataRequest, 
which is what I think when restarting my consumers so that they 
can begin working where they last left off (in the event that 
they were stopped for a while then restarted some time later, 
and new messages had come in).

The fetchOffsets() works on time, usually it looks like you send 
it Earliest or Latest (beginning or end of what's currently in 
the stream).

I realize the documentation says this:


 *Downsides of using SimpleConsumer*The SimpleConsumer does require a 
 significant amount of work not needed in the Consumer Groups:

1. You must keep track of the offsets in your application to know 
 where you left off consuming.

 But that's not really quite true ... not as long as commitOffsets() 
 has been provided.  It seems the SimpleConsumer provides you with a 
 solution to only one half of the problem of offset management.

Using some zookeeper python scripts I 

RE: Producer duplicates

[Arunkumar Srambikkal (asrambik)]

Thanks Jay. Any idea, when this feature can be expected? Are there any patches 
for it? 

Thanks
Arun 

-Original Message-
From: Jay Kreps [mailto:jay.kr...@gmail.com] 
Sent: Tuesday, February 17, 2015 10:16 PM
To: users@kafka.apache.org
Subject: Re: Producer duplicates

There are some details to get this right, the lookup table has to survive 
failures. But yes this is exactly what we would like to add:

https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer

-Jay

On Tue, Feb 17, 2015 at 12:44 AM, Arunkumar Srambikkal (asrambik)  
asram...@cisco.com wrote:

 Hi,

 I guess message production duplicate scenario in Kafka is, when a 
 producer commits the data but does not get an ack (broker or network 
 fails AFTER
 commit) and retries.

 I got thinking that the retry can be caught by the broker which could 
 then identify the previous message with a unique message ID before 
 committing them again but it requires a small lookup table for the 
 previous commit ID for each producer.

 Can I please know what is wrong about this picture, since Kafka does 
 not do this?

 Thanks
 Arun