Sharded Service Coordination

2014-07-09 Thread Oliver, James
Good morning, My company is in the process of adopting Apache's open source stack, and I've been tasked with building Aurora jobs to deploy a few popular open source technologies on Mesos. Aurora is an elegant scheduler and in our estimation will met our company's needs. However, we are struggl

Re: Sharded Service Coordination

2014-07-09 Thread Bill Farner
FWIW at Twitter we do something that sounds like a mix between (1) and (2). We consume the (otherwise unused) Announcer configuration parameter [1] to Job, and extend the executor to publish all allocated {{thermos.port[x]}}s to ZooKeeper. This is something we would love to open source, so please

Re: Sharded Service Coordination

2014-07-09 Thread Oliver, James
*Nudge nudge* We would love to take advantage of this tool if you folks would be so kind! Thanks, James On 7/9/14 1:35 PM, "Bill Farner" wrote: >FWIW at Twitter we do something that sounds like a mix between (1) and >(2). > We consume the (otherwise unused) Announcer configuration parameter [

Re: Sharded Service Coordination

2014-07-09 Thread Bhuvan Arumugam
Bill, I'm hoping we could also pull information like host, port, process, state, etc from WAL replica logs stored in scheduler. No? On Wed, Jul 9, 2014 at 1:35 PM, Bill Farner wrote: > FWIW at Twitter we do something that sounds like a mix between (1) and (2). > We consume the (otherwise unused)

Re: Sharded Service Coordination

2014-07-09 Thread Bill Farner
Unfortunately the mesos log API does not currently support reading the log from non-leading replicas. Depending on where we go with the storage overhaul, it's possible we could enable reads from non-leading schedulers, however. That said, querying this from the leading scheduler is relatively che