Hi Michael, So far none of our jobs have crashed yet and everything is working just fine, but I can already see a major memory saving (my estimate is at least 33% less memory is needed) with ThreadJobFactory.
One downside was that the logger wouldn't respect the lib/log4j.xml configuration file anymore, but that's probably some error on my part. Lukas On Sat, May 30, 2015 at 2:41 PM, Michael Ravits <michaelr...@gmail.com> wrote: > Hi Lukas, > > We use similar docker based deployment. > Would be glad if you could post here if ThreadJobFactory worked well for > you. > > Thanks, > Michael > > On Sat, May 30, 2015 at 12:22 AM, Lukas Steiblys <lu...@doubledutch.me> > wrote: > > > Yes, I think switching to ThreadJobFactory is a good solution. I think > the > > reasons why I switched to ProcessJobFactory earlier no longer hold true. > > > > Thanks. > > > > Lukas > > > > -----Original Message----- From: Yi Pan > > Sent: Friday, May 29, 2015 2:05 PM > > > > To: dev@samza.apache.org > > Subject: Re: ProcessJobFactory parent process > > > > Hi, Lukas, > > > > Yes. That's exactly part of the feature to allow > > health-check/failure-detection of containers. Another short-term option > is > > trying to use ThreadJobFactory, which has the JobCoordinator and > containers > > in the same process. Does that work for your use case? > > > > -Yi > > > > On Fri, May 29, 2015 at 12:59 PM, Lukas Steiblys <lu...@doubledutch.me> > > wrote: > > > > Yes, I'm talking about the child process crashing. I'd like the parent > to > >> die as well if the child crashes so Docker can understand that the > process > >> failed and restart the container. > >> > >> Lukas > >> > >> -----Original Message----- From: Yi Pan > >> Sent: Friday, May 29, 2015 12:47 PM > >> To: dev@samza.apache.org > >> Subject: Re: ProcessJobFactory parent process > >> > >> > >> Hi, Lukas, > >> > >> I assume that when you say "the job crashes", you were referring to the > >> child process running the container, not the parent process? If yes, we > >> were actually talking about adding container > >> health-check/failure-detection > >> in the JobCoordinator. SAMZA-680 would be the good place to start these > >> kind of discussion. > >> > >> Thanks! > >> > >> -Yi > >> > >> On Fri, May 29, 2015 at 8:39 AM, Lukas Steiblys <lu...@doubledutch.me> > >> wrote: > >> > >> Hi Yan, > >> > >>> > >>> The memory usage is not very high, but I'm trying to cut the usage any > >>> way > >>> I can. > >>> > >>> The bigger problem is when the job crashes and the parent process stays > >>> active preventing an auto restart by the Docker supervisor. > >>> > >>> Lukas > >>> > >>> On Thursday, May 28, 2015, Yan Fang <yanfang...@gmail.com> wrote: > >>> > >>> > Hi Lukas, > >>> > > >>> > The parent process is used to manage the lifecycle of the actual > >>> process. I > >>> > am curious how much memory the parent process takes? > >>> > > >>> > Thanks, > >>> > > >>> > Fang, Yan > >>> > yanfang...@gmail.com <javascript:;> > >>> > > >>> > On Thu, May 28, 2015 at 2:30 PM, Lukas Steiblys < > lu...@doubledutch.me > >>> > <javascript:;>> > >>> > wrote: > >>> > > >>> > > Hello, > >>> > > > >>> > > I’m running Samza tasks using ProcessJobFactory and after I start > the > >>> > job, > >>> > > the initial process spawns a new process that is the actual process > >>> where > >>> > > the code is run. The problem is that the parent process stays > active > >>> even > >>> > > after the job is started and that messes with the way I deploy > Samza > >>> (in > >>> > > Docker containers) and consumes memory while not doing anything. > >>> > > > >>> > > My question: is it possible to kill the parent process while still > >>> > leaving > >>> > > the Samza tasks to process messages? > >>> > > > >>> > > Lukas > >>> > > >>> > >>> > >>> > >> > > >
