Re: Supervisord for Kafka 0.8.1
Thanks guys. Will try these two.
On Mar 25, 2016 12:36, "Daniel Tamai" <daniel.ta...@gmail.com> wrote:
> Hi Kashyap,
>
> Using supervisord 3.2, this is the snippet from my supervisord.conf for
> Kafka:
>
> [program:kafka]
> command=%(ENV_KAFKA_HOME)s/bin/kafka-server-start.sh
> %(ENV_CONFIG_DIR)s/server.properties
> stdout_logfile=%(ENV_LOG_DIR)s/kafka_out.log
> stderr_logfile=%(ENV_LOG_DIR)s/kafka_err.log
> umask=022
> autostart=false
> autorestart=true
> startsecs=10
> startretries=3
>
> I don't think this will work with supervisord 3.1.
>
> Em sex, 25 de mar de 2016 às 13:08, Achanta Vamsi Subhash <
> achanta.va...@flipkart.com> escreveu:
>
> > We use daemontools and this is our run file:
> >
> > #!/bin/bash
> >
> > PAC=kafka-0.8.2.x
> > APP_HOME=/usr/share/$PAC
> > # app options
> > APP_CONFIG_HOME=${APP_HOME}/config
> > APP_OPTS="${APP_CONFIG_HOME}/server.properties"
> > JVM_OPTS=""
> >
> > # jvm user options
> > if [ "abc$KAFKA_HEAP_OPTS" == "abc" ]; then
> > export KAFKA_HEAP_OPTS="-Xms8g -Xmx8g"
> > fi
> >
> > if [ "abc$KAFKA_JVM_PERFORMANCE_OPTS" == "abc" ]; then
> > export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC
> > -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
> > -Djava.awt.headless=true"
> > fi
> >
> > exec setuidgid $APP_USER $APP_HOME/bin/kafka-server-start.sh $JVM_OPTS
> > $APP_OPTS 2>&1
> >
> >
> > On Fri, Mar 25, 2016 at 8:13 PM, Kashyap Mhaisekar <kashya...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > Am having trouble configuring Kafka server starts with supervisord.
> > >
> > > Has anyone from this group succeeded in integrating Kafka server start
> > and
> > > stop via supervisord? Can you please share the snippet of his out of
> > > configured?
> > >
> > > Thanks
> > > Kashyap
> > >
> >
> >
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
Supervisord for Kafka 0.8.1
Hi, Am having trouble configuring Kafka server starts with supervisord. Has anyone from this group succeeded in integrating Kafka server start and stop via supervisord? Can you please share the snippet of his out of configured? Thanks Kashyap
Re: Keeping Zookeeper and Kafka Server Up
We use supervisord for this. It ensures that the processes are always up and running. Thanks Kashyap On Wednesday, June 17, 2015, Shayne S shaynest...@gmail.com wrote: kafka-server-start.sh has a -daemon option, but I don't think Zookeeper has it. On Tue, Jun 16, 2015 at 11:32 PM, Su She suhsheka...@gmail.com javascript:; wrote: It seems like nohup has solved this issue, even when the putty window becomes inactive the processes are still running (I din't need to interact with them). I might look into using screen or tmux as a long term solution. Thanks Terry and Mike! Best, Su On Tue, Jun 16, 2015 at 3:42 PM, Terry Bates terryjba...@gmail.com javascript:; wrote: Greetings, nohup does the trick, as Mr. Bridge has shared. If you seem to want to run these and still have some interactivity with the services, consider using screen or tmux as these will enable you to run these programs in foreground, have added windows you can use to access shell, tail logs, and so on, and enable you to disconnect from the session, but still have these sessions available for re-attachment. In addition, I using runit for service supervision may enable you to keep daemons running, but if your services are dying you may need to introspect more deeply on the root cause versus working around it by restarting them. *Terry Bates* *Email: *terryjba...@gmail.com javascript:; *Phone: (*412) 215-0881 *Skype*: terryjbates *GitHub*: https://github.com/terryjbates *Linkedin*: http://www.linkedin.com/in/terryjbates/ On Tue, Jun 16, 2015 at 3:30 PM, Mike Bridge m...@bridgecanada.com javascript:; wrote: Have you tried using nohup nohup bin/zookeeper-server-start.sh config/zookeeper.properties nohup bin/kafka-server-start.sh config/server.properties On Tue, Jun 16, 2015 at 3:21 PM, Su She suhsheka...@gmail.com javascript:; wrote: Hello Everyone, I'm wondering how to keep Zookeeper and Kafka Server up even when my SSH (using putty) becomes inactive. I've tried running it in the background (using ), but it seems like it stops sometimes after a couple hours or so and I'll have to restart zookeeper and/or the kafka server. The only remediation i've found is to export TMOUT=[big number], but there must be another solution. Thank you! Best, Su
Re: Security in 0.8.2 beta
Thanks. We use the encryption approach as well. But the 2 topic approach is unique. Thank you. Kashyap. On Nov 12, 2014 1:54 AM, Joe Stein joe.st...@stealth.ly wrote: I know a few implements that do this encrypt your messages with a PSK between producers and consumers. One of them actually writes the encrypted symmetric key on a different topic foreach downstream consumer private key that can read the message. This way when you are consuming you consume from two topics 1) the topic with the message (which is encrypted) you want 2) the topic that you can use your private key to decrypt (because your public key was used) the symmetric key and then use that to decrypt the message (which you join from the two streams by the uuid so each message has a different secrete key encrypted with your public key) The other ones I can't talk about =8^) but this one I mention is interesting solution to this problem with Kafka I really like. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Nov 12, 2014 at 2:41 AM, Mathias Herberts mathias.herbe...@gmail.com wrote: Simply encrypt your messages with a PSK between producers and consumers. On Nov 12, 2014 4:38 AM, Kashyap Mhaisekar kashya...@gmail.com wrote: Hi, Is there a way to secure the topics created in Kafka 0.8.2 beta? The need is to ensure no one is asked to read data from the topic without authorization. Regards Kashyap
Security in 0.8.2 beta
Hi, Is there a way to secure the topics created in Kafka 0.8.2 beta? The need is to ensure no one is asked to read data from the topic without authorization. Regards Kashyap
Partition reassign Kafka 0.8.1.1
Hi,
I was testing out Kafka 0.8.1.1 and found that i get the following
exception during partition re-assignment :
*./kafka-reassign-partitions.sh --path-to-json-file ritest.json --zookeeper
localhost:2181*
*Partitions reassignment failed due to Partition reassignment currently in
progress for Map(). Aborting operation*
*kafka.common.AdminCommandFailedException: Partition reassignment currently
in progress for Map(). Aborting operation*
*at
kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:91)*
*at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)*
*at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)*
*My JSOn file is as follows *
*{partitions:[{topic: ritest1,partition: 3,
replicas: [1,2,3] }]}*
What am I doing wrong?
Regards,
Kashyap
Re: Loss of Leader in Kafka
After topic deletion, I deleted the physical folders in data logs and then restarted Kafka. That resolved. Thanks! Will try out 0.8.1 and let the group know. Regards, Kashyap On Monday, May 12, 2014, Jun Rao jun...@gmail.com wrote: Delete topic doesn't quite work in 0.8.1. We recently fixed it in trunk. Could you give it a try and see if you see the same issue? Thanks, Jun On Mon, May 12, 2014 at 9:39 AM, Kashyap Mhaisekar kashya...@gmail.comjavascript:; wrote: Hi, I am hitting a strange exception while creating a topic in Kafka - Steps to generate this- 1. Created a topic multipartition_test with 2 partitions and 2 replicas 2. Added some data to this topics and verified data is coming up for both partitions 3. Deleted the topic. Checked only the zookeeper to see if the /brokers/topics DOES NOT have the topic 4. Recreated the topic in exactly the same way as in point 1. After this, when I list topics using ./kafka-list-topic.sh, i see that *leader:* none and *isr:* for this topic. State change logs give the following exception. kafka.common.StateChangeFailedException: encountered error while electing leader for partition [multipartition_test,1] due to: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state. at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:327) at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:154) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:110) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:109) at scala.collection.immutable.Set$Set2.foreach(Set.scala:101) at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:109) at kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:325) at kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:312) at kafka.controller.PartitionStateMachine$TopicChangeListener.liftedTree1$1(PartitionStateMachine.scala:376) at kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:361) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Caused by: kafka.common.StateChangeFailedException: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state at kafka.controller.PartitionStateMachine.getLeaderIsrAndEpochOrThrowException(PartitionStateMachine.scala:347) at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:291) ... 11 more Can you please help what am I doing wrong? Regards, kashyap
Loss of Leader in Kafka
Hi, I am hitting a strange exception while creating a topic in Kafka - Steps to generate this- 1. Created a topic multipartition_test with 2 partitions and 2 replicas 2. Added some data to this topics and verified data is coming up for both partitions 3. Deleted the topic. Checked only the zookeeper to see if the /brokers/topics DOES NOT have the topic 4. Recreated the topic in exactly the same way as in point 1. After this, when I list topics using ./kafka-list-topic.sh, i see that *leader:* none and *isr:* for this topic. State change logs give the following exception. kafka.common.StateChangeFailedException: encountered error while electing leader for partition [multipartition_test,1] due to: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state. at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:327) at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:154) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:110) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:109) at scala.collection.immutable.Set$Set2.foreach(Set.scala:101) at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:109) at kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:325) at kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:312) at kafka.controller.PartitionStateMachine$TopicChangeListener.liftedTree1$1(PartitionStateMachine.scala:376) at kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:361) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Caused by: kafka.common.StateChangeFailedException: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state at kafka.controller.PartitionStateMachine.getLeaderIsrAndEpochOrThrowException(PartitionStateMachine.scala:347) at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:291) ... 11 more Can you please help what am I doing wrong? Regards, kashyap
Re: Log Retention in Kafka
Thanks Jun. I am using kafka_2.8.0-0.8.0-beta1.jar. I dont see this script kafka-topic.sh in the bin folder. Is there way to do this in the version mentioned? --Kashyap On Mon, May 5, 2014 at 11:18 AM, Jun Rao jun...@gmail.com wrote: See http://kafka.apache.org/documentation.html#basic_ops_modify_topic Thanks, Jun On Sun, May 4, 2014 at 10:11 PM, Kashyap Mhaisekar kashya...@gmail.com wrote: Is there a way to do this at runtime using some available scripts in kafka/bin? If so, any pointers on which script? Regards, Kashyap On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar kashya...@gmail.com wrote: Thanks Joel. Am using version 2.8.0. Thanks, Kashyap On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy jjkosh...@gmail.com wrote: Which version of Kafka are you using? You can read up on the configuration options here: http://kafka.apache.org/documentation.html#configuration You can specify time-based retention using log.retention.minutes which will apply to all topics. You can override that on per-topic basis - see further down in the above page under topic-level configuration On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some queries around it.And it was mentioned that topic.log.retention.hours is per topic configuration. Had some queries around it - 1. Does it mean that I need to specific the topicname.log.retention.hours in the kafka config? 2. Can this property be overriden anywhere? 3. Is it possible for the producer to set a message expiry so that the message expires after a configurable period of time? Regards, Kashyap
Re: Log Retention in Kafka
Is there a way to do this at runtime using some available scripts in kafka/bin? If so, any pointers on which script? Regards, Kashyap On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar kashya...@gmail.comwrote: Thanks Joel. Am using version 2.8.0. Thanks, Kashyap On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy jjkosh...@gmail.com wrote: Which version of Kafka are you using? You can read up on the configuration options here: http://kafka.apache.org/documentation.html#configuration You can specify time-based retention using log.retention.minutes which will apply to all topics. You can override that on per-topic basis - see further down in the above page under topic-level configuration On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some queries around it.And it was mentioned that topic.log.retention.hours is per topic configuration. Had some queries around it - 1. Does it mean that I need to specific the topicname.log.retention.hours in the kafka config? 2. Can this property be overriden anywhere? 3. Is it possible for the producer to set a message expiry so that the message expires after a configurable period of time? Regards, Kashyap
Log Retention in Kafka
Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some queries around it.And it was mentioned that topic.log.retention.hours is per topic configuration. Had some queries around it - 1. Does it mean that I need to specific the topicname.log.retention.hours in the kafka config? 2. Can this property be overriden anywhere? 3. Is it possible for the producer to set a message expiry so that the message expires after a configurable period of time? Regards, Kashyap
Re: Log Retention in Kafka
Thanks Joel. Am using version 2.8.0. Thanks, Kashyap On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy jjkosh...@gmail.com wrote: Which version of Kafka are you using? You can read up on the configuration options here: http://kafka.apache.org/documentation.html#configuration You can specify time-based retention using log.retention.minutes which will apply to all topics. You can override that on per-topic basis - see further down in the above page under topic-level configuration On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some queries around it.And it was mentioned that topic.log.retention.hours is per topic configuration. Had some queries around it - 1. Does it mean that I need to specific the topicname.log.retention.hours in the kafka config? 2. Can this property be overriden anywhere? 3. Is it possible for the producer to set a message expiry so that the message expires after a configurable period of time? Regards, Kashyap
Leader: -1 on Kafka
Hi, At times, some of kafka topics end up showing the leader as -1. After this, the messages dont get added to the topic nor consumed. I tried digging into why the leader turns -1 *(leader: -1)* Is there a reason why this happens and how it can be resolved? Regards, kashyap
Re: Leader: -1 on Kafka
No. Is this error common? How to overcome this? Regards, Kashyap On Mon, Apr 21, 2014 at 4:02 PM, Joel Koshy jjkosh...@gmail.com wrote: So only one of the brokers have the data? i.e., with one partition. Do you see errors in your broker log around that time? What do you see in the kafka-state-change.log for that topic/partition? On Mon, Apr 21, 2014 at 03:36:52PM -0500, Kashyap Mhaisekar wrote: Yes. I have 3 kafka brokers and I have created one topic with 1 partition and 1 replication (default options). All params are default. I was testing it with heavy load and had a storm reading from this and suddenly the messages stop. kafka's list topics script files then show the leader as -1. I got the same error even with 2 partitions and default replication. Regards, Kashyap On Mon, Apr 21, 2014 at 1:26 PM, Joel Koshy jjkosh...@gmail.com wrote: Can you describe your set up in more detail and also if you can reproduce this easily? This can happen when none of the replicas for a partition are available, but cannot comment further without details. On Mon, Apr 21, 2014 at 10:54:59AM -0500, Kashyap Mhaisekar wrote: Hi, At times, some of kafka topics end up showing the leader as -1. After this, the messages dont get added to the topic nor consumed. I tried digging into why the leader turns -1 *(leader: -1)* Is there a reason why this happens and how it can be resolved? Regards, kashyap
