Thanks everyone for all the help. Marathon does seem like a good framework but my use case requires the app to evaluate it's own health and scale up based on internal load stats (SLA requirements) and I don't know if marathon supports that. This is the main reason why i am looking at building out my own scheduler/executor. I will give another go with Vinod's comments and have a look at the hadoop scheduler.
Just a recommendation to any mesos experts out there, it would be super helpful if there was a complete mock app with annotated code somewhere. Another good page on the website would a good FAQ page. I am still pretty n00b as far as mesos is concerned, so, pardon any stupid comments/suggestions/questions. -- Ankur On Fri, Dec 27, 2013 at 10:16 AM, Abhishek Parolkar <[email protected]>wrote: > @Ankur, > In case Marathon looks like direction you want to go with, I have a > small demo in here if that helps > http://www.youtube.com/watch?v=2YWVGMuMTrg > > -parolkar > > > On Sat, Dec 28, 2013 at 2:10 AM, Vinod Kone <[email protected]> wrote: > >> I can't really find an example that is an end-to-end use case. By that I >>> mean, I would like to know how to put the scheduler and the executor in the >>> correct places. Right now I have a single jar with can be run from the >>> command line: java -jar target/collector.jar and that would take care of >>> everything. >>> >>> This collector.jar can act as both scheduler and executor, presumably >> based on command line flags? If yes, thats certainly doable. Typically the >> scheduler and executor are split into separate jars. This makes it easy to >> decouple the upgrade of scheduler and executor. >> >> >>> My current train of thought is that the webapp jar would stay somewhere >>> on an S3 url and the "CollectorScheduler" would "somehow" tell a mesos >>> slave to run the "CollectorExecutor" which in turn fetch the jar from S3 >>> and run it. >>> >>> >> Yes, you are on the right track. Mesos slave can download the jar for you >> as long as it could be accessed via (http://, https://, ftp://, hdfs:// >> etc). This is how you do it: >> >> When you launch a task from the scheduler via 'launchTaks()' you give it >> a 'vector<TaskInfo>' as one of the arguments. Since you are using a custom >> executor you should set 'TaskInfo.ExecutorInfo' (see mesos.proto) to point >> to your executor. To specify the S3 URL you would set >> 'TaskInfo.ExecutorInfo.CommandInfo.URI.value'. To tell slave the command to >> launch the executor after it downloads the the executor, you would set >> 'TaskInfo.ExecutorInfo.CommandInfo.value'. >> >> You can find some examples here: >> >> Hadoop >> scheduler<https://github.com/mesos/hadoop/blob/master/src/main/java/org/apache/hadoop/mapred/ResourcePolicy.java> >> >> Example Java scheduler >> <https://github.com/apache/mesos/blob/master/src/examples/java/TestFramework.java> >> >> Hope that helps. Let us know if you have additional questions. >> >> Vinod >> > >
