On Monday, 16 October 2017 01:53:25 UTC+1, Alex Buchanan wrote: > > Not a spanner at all. > > I think the Task Execution Schemas (TES) [1], which Funnel is based on, is > a reinvention of DRMAA using technologies such as HTTP, REST, JSON, > Protobuf. It's a pretty simple API and message type (Task) for create, get, > list, cancel. But, admittedly, I don't know enough about DRMAA. I get a bit > overwhelmed by its documentation, to be honest. >
Yea the DRMMA documentation is quite overwhelming, I think it was written a long time ago by Sun/SGI users AFAIK. > > Funnel is an implementation of the TES spec. We'd like to keep it > versatile for the reason you mentioned; many solutions end up feeling > heavyweight and you get pigeon holed. We also think a lot about workflows, > and the status quo is similar there. Funnel already supports many > environments and schedulers: GCE, AWS, HTCondor, SGE, etc, etc. > We're talking about adding Kubernetes. We're always thinking of ways to > make it easier and more flexible, hence the thoughts about how far we > should take the scheduler. If you need to run 10K tasks in a new GCE > project on preemptible machines, how easy can we make that? > Having tried using things like gleam/glow (see https://github.com/chrislusf/gleam) and dask (which has similar ideas but written in python) it sounds like a FIFO work stealing type scheduler wouldn't be a bad idea. In particular dask's approach is quite nicely put together. Though one thing that I have noticed with a lot of these non-HPC type schedulers is that they are great for doing embarrassingly parallel type problems but fall apart reasonably quickly once you get into MPI territory where you want to be careful about how you layout the processes/network for your job, probably things to consider when writing a scheduler. > > I encourage you to take a look at Funnel, let me know what you think. If > it's lacking something that you need, I'd be interested in hearing about it. > Will do, this looks interesting and may address some of my needs for wanting an web API on top of Slurm. > > Anywho, I'll stop ranting now. Thanks for the feedback! > > -Alex > > [1] https://github.com/ga4gh/task-execution-schemas > > > On Sunday, October 15, 2017 at 12:00:03 PM UTC-7, Jimmy Tang wrote: >> >> Not to throw a spanner into the works, but we have a similar problem in >> my work environment of needing a scheduler to schedule distributed jobs, >> one problem of writing a *nice* one for a given language is that you end up >> being pigeon holed into one solution. We've been looking at using drmaa as >> a way of accessing different schedulers in a more platform and language >> agnostic way. It may be worth your while to take a look at the golang >> bindings for drmaa so you aren't left reinventing the wheel. Maybe creating >> a dumb scheduler for drmaa library might be the way to go? >> >> On Wednesday, 11 October 2017 20:14:46 UTC+1, Alex Buchanan wrote: >>> >>> Hey all, >>> >>> In Funnel (a distributed task toolkit) we're sort of dancing around >>> having a full-on scheduler. We have a scheduler that has grown from >>> development util, to prototype, to something we actually use, but it's >>> missing many of the features you'd want in production. Mostly we aim to >>> delegate scheduling to another application (SGE, Slurm, AWS Batch, >>> Kubernetes, etc), but having a built-in ability to schedule tasks without >>> extra infrastructure is undeniably attractive. >>> >>> Writing a scheduler is one of those things people warn you away from >>> though. I wish there was a solid library we could embed, but I haven't >>> found anything. >>> >>> I wanted to get some opinions from this community. Do you know of any >>> scheduling libraries? Do you think having scheduling built in is a good >>> idea? A bad idea? Should we keep chipping away at it? Would people be >>> interested in a standalone scheduling library, or is this problem >>> inherently too complex to be adequately captured in library form? >>> >>> Thanks! >>> >>> Funnel: https://github.com/ohsu-comp-bio/funnel >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
