Re: ThreadJobFactory in production

Thanks for the clarification -- very helpful. I'll take a look at those tickets! On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan wrote: > Hi, Robert, > > The main reason that ThreadJobFactory and ProcessJobFactory are not > considered "production-ready" is that there is only one

Re: Review Request 44293: SAMZA-883 Improve logging for container handling and kafka abdication

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44293/#review121776 --- Ship it! +1 from my side! lgtm - Jagadish Venkatraman On

Re: Lag metric

Never mind, I forgot to "listen" to new metrics in my reporter, so those metrics which were created immediately worked fine, and those which were created later, for example when kafka metadata is fetched and count of partitions is discovered, were ignored by my reporter. On Wed, Mar 2, 2016 at

Lag metric

Hi all, I want to report my task's kafka topic lag metric. I have implemented my own MetricReporterFactory to report to Influxdb. I can see that KafkaSystemConsumerMetrics class register "messages-behind-high-watermark" metric, but I can not trace where would it be registered with the reporter. I

Re: ThreadJobFactory in production

Jose, It would be great if you could share it. I'm interested in trying to use it as well. Thanks, Roger On Wed, Mar 2, 2016 at 2:31 PM, José Barrueta wrote: > Hi guys, > > At Stormpath, we made a custom samza 10 version merging SAMZA-41 into it, > it's working well, so

Re: ThreadJobFactory in production

Hi Jose, Thanks for your inputs! It would be great if you could give your inputs on SAMZA-881. SAMZA-881 ensures that we make it easy to integrate with resource managers. If you could add a comment there on how you are currently using (a patched version) of Samza, we 'd love to align that with

Re: ThreadJobFactory in production

Hi guys, At Stormpath, we made a custom samza 10 version merging SAMZA-41 into it, it's working well, so we are thinking to update that patch later this week so it can be added to the main project. HTH, Jose Luis Barrueta On Wed, Mar 2, 2016 at 2:11 PM, Yi Pan wrote: >

Re: Review Request 44241: SAMZA-883 Improve logging for container handling and kafka refresh

> On March 2, 2016, 10:20 p.m., Boris Shkolnik wrote: > > Looks like this debug->info conversions are mainly for container > > allocations. So it should not add too much noise and should be quite > > helpfull. > > Navina Ramesh wrote: > @Boris: This change has been discarded. Please check

Re: Review Request 44241: SAMZA-883 Improve logging for container handling and kafka refresh

> On March 2, 2016, 10:20 p.m., Boris Shkolnik wrote: > > Looks like this debug->info conversions are mainly for container > > allocations. So it should not add too much noise and should be quite > > helpfull. @Boris: This change has been discarded. Please check the new RB here -

Re: Review Request 44241: SAMZA-883 Improve logging for container handling and kafka refresh

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44241/#review121736 --- Ship it! Looks like this debug->info conversions are mainly

Re: Review Request 44293: SAMZA-883 Improve logging for container handling and kafka abdication

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44293/#review121734 --- Ship it! lgtm. Please check w/ Jon regarding to the WARN/INFO

Re: ThreadJobFactory in production

Hi, Robert, The main reason that ThreadJobFactory and ProcessJobFactory are not considered "production-ready" is that there is only one container for the job and all tasks are assigned to the single container. Hence, it is not easy to scale out of a single host. As Rick mentioned, Netflix has

Re: checkpoint example?

Let me explain the use case in more detail: We are keeping the data in 2 systems in sync. Let's name the upstream system as the "source" and the downstream system as "destination". The destination system is backed up (locally) once per day (let's say at 3:00 AM). It's now 1:00 PM, and we've

Re: ThreadJobFactory in production

There was an interesting thread a while back from I believe the netflix guys about running ThreadJobFactory in production. > On Mar 2, 2016, at 4:20 PM, Robert Crim wrote: > > Hi, > > We're currently working on a solution that allows us to run Samza jobs on > Mesos. This

ThreadJobFactory in production

Hi, We're currently working on a solution that allows us to run Samza jobs on Mesos. This seems to be going well, and something we'd like to move away from when native Mesos support is added to Samza. While we're developing and testing our scheduler, I'm wondering about the implications of

Re: [DISCUSS] Re-thinking the Samza Job Coordinator

This is awesome! Samza will be able to run standalone or on different kinds of clusters, which can serve a wide range of applications. Glad to see this is happening. Thanks, Xinyu On Tue, Mar 1, 2016 at 8:48 AM, Jagadish Venkatraman wrote: > Hi all, > > Currently, the

Re: Review Request 43732: Implemented AvroDataFileHdfsWriter

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43732/ --- (Updated March 2, 2016, 4:35 p.m.) Review request for samza. Changes ---