Hi guys, At Stormpath, we made a custom samza 10 version merging SAMZA-41 into it, it's working well, so we are thinking to update that patch later this week so it can be added to the main project.
HTH, Jose Luis Barrueta On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan <nickpa...@gmail.com> wrote: > Hi, Robert, > > The main reason that ThreadJobFactory and ProcessJobFactory are not > considered "production-ready" is that there is only one container for the > job and all tasks are assigned to the single container. Hence, it is not > easy to scale out of a single host. > > As Rick mentioned, Netflix has put up a patch in SAMZA-41 based on 0.9.1 o > allow static assignment of a subset of partitions to a single ProcessJob, > which allows to launch multiple ProcessJobs in different hosts. We planned > to merge it to 0.10. But it turns out that too much changes have gone into > 0.10 and it became difficult to merge the patch. At this point, we can > still try the following two options: > 1) We can attempt to merge SAMZA-41 to 0.10.1 again, it may take some > effort but would give a stop-gap solution. > 2) We are working on a standalone Samza model (SAMZA-516, SAMZA-881) to > allow users to run Samza w/o depending on YarnJobFactory. This is a > long-term effort and will take some time to flesh out. Please join the > discussion there s.t. we can be more aligned in our effort. > > Hope the above gives you an overall picture on where we are going. > > Thanks a lot! > > -Yi > > On Wed, Mar 2, 2016 at 1:28 PM, Rick Mangi <r...@chartbeat.com> wrote: > > > There was an interesting thread a while back from I believe the netflix > > guys about running ThreadJobFactory in production. > > > > > > > On Mar 2, 2016, at 4:20 PM, Robert Crim <rjc...@gmail.com> wrote: > > > > > > Hi, > > > > > > We're currently working on a solution that allows us to run Samza jobs > on > > > Mesos. This seems to be going well, and something we'd like to move > away > > > from when native Mesos support is added to Samza. > > > > > > While we're developing and testing our scheduler, I'm wondering about > the > > > implications of running tasks with the ThreadJobFactory in > "production". > > > The documentation advise against this, but it's not clear why. > > > > > > If we were using the ThreadJobFactory inside of a docker container on > > Mesos > > > with Marathon for production, would be our main problem? These are not > > > particularly high-load tasks. Aside from not be able to get > find-grained > > > resource scheduling per-task, it seems like the main issue the not > being > > to > > > easily tell when a job stops due to error / exception. > > > > > > In other words, what would be stop-stopping reasons to not use the > > > TreadJobFactory in production? > > > > > > Thanks, > > > Rob > > > > >
