Hello Guozhang and Akshat, We are doing the SIGTERM / shutdown hook in our Kafka streams. We will write a blog post when we are done. Thanks for everything help.
Phillp From: Guozhang Wang <wangg...@gmail.com> Date: Tuesday, July 19, 2016 at 2:06 PM To: "users@kafka.apache.org" <users@kafka.apache.org> Cc: Phillip Mann <pm...@trulia.com> Subject: Re: Deploying Kafka Streams to production (Ubuntu 16.04 / systemd) Hi Phillip, You need to add the shutdown hook yourself when coding your application using the Kafka Streams library. We are updating our web documentations to add an example about how to do that, but generally speaking it will be similar to what the Spark example shows you. Guozhang On Mon, Jul 18, 2016 at 12:05 PM, Akshat Aranya <aara...@gmail.com<mailto:aara...@gmail.com>> wrote: This is something that perhaps someone else can comment on; I haven't played Kafka Streams yet. That said, your Kafka Streams application would need to shut down gracefully, whether you want to shut it down via Ctrl-C or use systemd :-). -Akshat On Mon, Jul 18, 2016 at 11:40 AM, Phillip Mann <pm...@trulia.com<mailto:pm...@trulia.com>> wrote: > Hi Akshat, > > > > Do you know if a Kafka Streams apps can gracefully handle SIGTERM or do we > have to implement it similarly to here: > https://metabroadcast.com/blog/stop-your-spark-streaming-application-gracefully > > > > Thanks again for your help. > > > > Phillip > > > > *From: *Akshat Aranya <aara...@gmail.com<mailto:aara...@gmail.com>> > *Date: *Monday, July 18, 2016 at 10:12 AM > *To: *"users@kafka.apache.org<mailto:users@kafka.apache.org>" > <users@kafka.apache.org<mailto:users@kafka.apache.org>> > *Cc: *Phillip Mann <pm...@trulia.com<mailto:pm...@trulia.com>> > *Subject: *Re: Deploying Kafka Streams to production (Ubuntu 16.04 / > systemd) > > > > Hi Phillip, > > In general, it is trivial to take an existing java -jar command and > systemd-ify it. All you need to do is to run it as an ExecStart in the > foreground in the systemd unit file, possibly with a User setting. > Something like this should work: > > [Unit] > Description=My stream app > > [Service] > User=unprivileged_user > Restart=always > ExecStart=/usr/bin/java -jar ... > > [Install] > WantedBy=multi-user.target > > systemd will send a SIGTERM to the java process by default when you invoke > systemctl stop. As a bonus, you also get log management of stdout/stderr > via journald. > > > > On Mon, Jul 18, 2016 at 9:08 AM, Phillip Mann > <pm...@trulia.com<mailto:pm...@trulia.com>> wrote: > > Hello Guozhang, > > Thanks for your reply. I am interested in systemd because that is the > primary method for the environment we are working with (AWC EC2 Ubuntu > 16.04 / systemd) at my company and on this project. If there is no way to > deploy with systemd, then we can explore alternatives. Hopefully there is > a way to do so with systemd ☺. Thanks again for your help! > > Phillip > > Hello Phillip, > > Thanks for your question. We are indeed working on write some guidance in > docs / blogs about how to deploy a written Kafka Streams app code with > different tools / frameworks. Are you particularly interested in systemd > because you are using it in your company, or are you interested in general > and OK with certain popular tools like YARN / Mesos / Chef / etc? > > > Guozhang > > > On Fri, Jul 15, 2016 at 3:57 PM, Phillip Mann > <pm...@trulia.com<mailto:pm...@trulia.com>> wrote: > > > Hello, > > > > There is currently no documentation how to deploy Kafka Streams to > > production. We have built a cool Kafka Streams application and we wish > to > > make it a part of our production system. At the moment, we currently run > > the Kafka streaming application with the java –jar command and we stop > the > > process with the kill command (ctrl-c). Obviously, this is not how we > > should deploy this application to production ☺. I have searched and > > searched and found no documentation, blog, question, anything about how > to > > deploy a Kafka Streams jar to production. In the interest of this > thread, > > I am specifically interested in deploying our Kafka Streams app to Ubuntu > > 16.04 and use systemd to start, stop, and if possible check the status of > > the JVM process itself. It would be something similar to how Spring Boot > > works ( > > > http://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html > ) > > with Systemd. Please let me know what options are available for Kafka > > Streams deployment and if systemd is possible. Thanks for your help and > > great job on Kafka Streams! > > > > Phillip > > > > > -- > -- Guozhang > > > -- -- Guozhang
