Diego- You might want to look at this project for some insights:
https://github.com/mesosphere/RENDLER -John On Wed, Feb 25, 2015 at 5:27 PM, Diego Medina <[email protected]> wrote: > Hi, > > > Short: Is it better to have one big framework and executor with if > statements to select what to do or several smaller framework <-> executors > when writing a Mesos app? > > Longer question: > > Last week I started a side project based on mesos (using Go), > > http://blog.fmpwizard.com/blog/web-crawler-using-mesos-and-golang > https://github.com/fmpwizard/owlcrawler > > It's a web crawler written on top of Mesos, The very first version of it > had a framework that sent a task to an executor and that single executor > would fetch the page, extract links from the html and then send them to a > message queue. > > Then the framework reads that queue and starts again, run the executor, > etc, etc. > > Now I'm splitting fetching the html and extracting links into two separate > tasks, and putting those two tasks in the same executor doesn't feel right, > so I'm thinking that I need at least two diff executors and one framework, > but then I wonder if people more experienced with mesos would normally > write several pairs of framework <-> executors to keep the design cleaner. > > On this particular case, I can see the project growing into even more > tasks that can be decoupled. > > Any feedback on the design would be great and let me know if I should > explain this better. > > Thanks > > > Diego > > > > > > -- > Diego Medina > Lift/Scala consultant > [email protected] > http://fmpwizard.telegr.am >
