Re: Executing Samza jobs natively in Kubernetes

Hi Elias, Thank you so much for your explanation. I'm looking into improving Samza Standalone and your design-inputs were very useful. Cheers, Jagadish On Mon, Dec 14, 2015 at 12:47 PM, Elias Levy wrote: > On Mon, Dec 14, 2015 at 11:55 AM, Jagadish Venkatraman < > jagadish1...@gmail.com> wrote

Re: Executing Samza jobs natively in Kubernetes

On Mon, Dec 14, 2015 at 11:55 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > > Thanks for the great work! This is super-helpful. Another cool feature is > that this implementation pushes alot of failure handling/restarting to the > cluster manager. > > I've some questions on the imple

Re: Executing Samza jobs natively in Kubernetes

Hey Elias, Thanks for the great work! This is super-helpful. Another cool feature is that this implementation pushes alot of failure handling/restarting to the cluster manager. I've some questions on the implementation. 1. It's my understanding that each container is a separate pod. When a pod c

Re: Executing Samza jobs natively in Kubernetes

Hi, Elias, Thanks a lot to put up the patch for the simple job running in Kubernetes! As Kartik mentioned, that is well aligned w/ our goal to make Samza job launching easier. I am glad that we actually share a lot of common ideas from independent minds. Let me try to give my opinions on this: 1.

Re: Executing Samza jobs natively in Kubernetes

I was too critical in my last comment on StandaloneJob. More specifically, StandaloneJob did not aid much on my implementation, which is based on the current 0.9.1 model of static a priori partition assignment. Within that model all that is needed is a command that takes in the job config and spi

Re: Executing Samza jobs natively in Kubernetes

Thanks for starting this discussion and great to see the integration with Kubernetes. It is very timely because we(at LinkedIn) have recently started to refocus on Samza-516 and were thinking of kicking off a similar discussion soon. I am in agreement with you that there are two possible levels t

Re: Executing Samza jobs natively in Kubernetes

BTW, I reviewed the StandaloneJob proposal in SAMZA-516 thinking it could have been useful in running Samza in Kubernetes and I was disappointed to find out it was not the case. StandaloneJob goes a step beyond the most basic job implementation as it attempts to handle failures through rebalances m

Re: Executing Samza jobs natively in Kubernetes

On Mon, Nov 30, 2015 at 12:31 PM, Jakob Homan wrote: > This is awesome work. Thanks. Feel like a hack, but it got the job done. > Would be interested in opening JIRAs for the > changes you need so we can start to process them? > I've opened: - SAMZA-824

Re: Executing Samza jobs natively in Kubernetes

Hey Elias- This is awesome work. Would be interested in opening JIRAs for the changes you need so we can start to process them? Thanks, Jakob On 30 November 2015 at 12:18, Roger Hoover wrote: > Awesome. Thanks. > > On Sun, Nov 29, 2015 at 3:25 PM, Elias Levy > wrote: > >> Roger, >> >> You a

Re: Executing Samza jobs natively in Kubernetes

Awesome. Thanks. On Sun, Nov 29, 2015 at 3:25 PM, Elias Levy wrote: > Roger, > > You are welcomed. If you want to experiment, you can use my hello samza > Docker image. > > On Sun, Nov 29, 2015 at 12:19 PM, Roger Hoover > wrote: > > > Elias, > > >

Re: Executing Samza jobs natively in Kubernetes

Roger, You are welcomed. If you want to experiment, you can use my hello samza Docker image. On Sun, Nov 29, 2015 at 12:19 PM, Roger Hoover wrote: > Elias, > > I would also love to be able to deploy Samza on Kubernetes with dynamic > task managemen

Re: Executing Samza jobs natively in Kubernetes

Elias, I would also love to be able to deploy Samza on Kubernetes with dynamic task management. Thanks for sharing this. It may be a good interim solution. Roger On Sun, Nov 29, 2015 at 11:18 AM, Elias Levy wrote: > I've been exploring Samza for stream processing as well as Kubernetes as a >

Executing Samza jobs natively in Kubernetes

I've been exploring Samza for stream processing as well as Kubernetes as a container orchestration system and I wanted to be able to use one with the other. The prospect of having to execute YARN either along side or on top of Kubernetes did not appeal to me, so I developed a KubernetesJob impleme