Hi, Jose, Great! I am looking forward to the patch! Could you open an RB to the JIRA?
Thanks a lot! -Yi On Wed, Mar 2, 2016 at 2:31 PM, José Barrueta <j...@stormpath.com> wrote: > Hi guys, > > At Stormpath, we made a custom samza 10 version merging SAMZA-41 into it, > it's working well, so we are thinking to update that patch later this week > so it can be added to the main project. > > HTH, > > Jose Luis Barrueta > > On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > Hi, Robert, > > > > The main reason that ThreadJobFactory and ProcessJobFactory are not > > considered "production-ready" is that there is only one container for the > > job and all tasks are assigned to the single container. Hence, it is not > > easy to scale out of a single host. > > > > As Rick mentioned, Netflix has put up a patch in SAMZA-41 based on 0.9.1 > o > > allow static assignment of a subset of partitions to a single ProcessJob, > > which allows to launch multiple ProcessJobs in different hosts. We > planned > > to merge it to 0.10. But it turns out that too much changes have gone > into > > 0.10 and it became difficult to merge the patch. At this point, we can > > still try the following two options: > > 1) We can attempt to merge SAMZA-41 to 0.10.1 again, it may take some > > effort but would give a stop-gap solution. > > 2) We are working on a standalone Samza model (SAMZA-516, SAMZA-881) to > > allow users to run Samza w/o depending on YarnJobFactory. This is a > > long-term effort and will take some time to flesh out. Please join the > > discussion there s.t. we can be more aligned in our effort. > > > > Hope the above gives you an overall picture on where we are going. > > > > Thanks a lot! > > > > -Yi > > > > On Wed, Mar 2, 2016 at 1:28 PM, Rick Mangi <r...@chartbeat.com> wrote: > > > > > There was an interesting thread a while back from I believe the netflix > > > guys about running ThreadJobFactory in production. > > > > > > > > > > On Mar 2, 2016, at 4:20 PM, Robert Crim <rjc...@gmail.com> wrote: > > > > > > > > Hi, > > > > > > > > We're currently working on a solution that allows us to run Samza > jobs > > on > > > > Mesos. This seems to be going well, and something we'd like to move > > away > > > > from when native Mesos support is added to Samza. > > > > > > > > While we're developing and testing our scheduler, I'm wondering about > > the > > > > implications of running tasks with the ThreadJobFactory in > > "production". > > > > The documentation advise against this, but it's not clear why. > > > > > > > > If we were using the ThreadJobFactory inside of a docker container on > > > Mesos > > > > with Marathon for production, would be our main problem? These are > not > > > > particularly high-load tasks. Aside from not be able to get > > find-grained > > > > resource scheduling per-task, it seems like the main issue the not > > being > > > to > > > > easily tell when a job stops due to error / exception. > > > > > > > > In other words, what would be stop-stopping reasons to not use the > > > > TreadJobFactory in production? > > > > > > > > Thanks, > > > > Rob > > > > > > > > >
