Not a spanner at all. I think the Task Execution Schemas (TES) [1], which Funnel is based on, is a reinvention of DRMAA using technologies such as HTTP, REST, JSON, Protobuf. It's a pretty simple API and message type (Task) for create, get, list, cancel. But, admittedly, I don't know enough about DRMAA. I get a bit overwhelmed by its documentation, to be honest.
Funnel is an implementation of the TES spec. We'd like to keep it versatile for the reason you mentioned; many solutions end up feeling heavyweight and you get pigeon holed. We also think a lot about workflows, and the status quo is similar there. Funnel already supports many environments and schedulers: GCE, AWS, HTCondor, SGE, etc, etc. We're talking about adding Kubernetes. We're always thinking of ways to make it easier and more flexible, hence the thoughts about how far we should take the scheduler. If you need to run 10K tasks in a new GCE project on preemptible machines, how easy can we make that? I encourage you to take a look at Funnel, let me know what you think. If it's lacking something that you need, I'd be interested in hearing about it. Anywho, I'll stop ranting now. Thanks for the feedback! -Alex [1] https://github.com/ga4gh/task-execution-schemas On Sunday, October 15, 2017 at 12:00:03 PM UTC-7, Jimmy Tang wrote: > > Not to throw a spanner into the works, but we have a similar problem in my > work environment of needing a scheduler to schedule distributed jobs, one > problem of writing a *nice* one for a given language is that you end up > being pigeon holed into one solution. We've been looking at using drmaa as > a way of accessing different schedulers in a more platform and language > agnostic way. It may be worth your while to take a look at the golang > bindings for drmaa so you aren't left reinventing the wheel. Maybe creating > a dumb scheduler for drmaa library might be the way to go? > > On Wednesday, 11 October 2017 20:14:46 UTC+1, Alex Buchanan wrote: >> >> Hey all, >> >> In Funnel (a distributed task toolkit) we're sort of dancing around >> having a full-on scheduler. We have a scheduler that has grown from >> development util, to prototype, to something we actually use, but it's >> missing many of the features you'd want in production. Mostly we aim to >> delegate scheduling to another application (SGE, Slurm, AWS Batch, >> Kubernetes, etc), but having a built-in ability to schedule tasks without >> extra infrastructure is undeniably attractive. >> >> Writing a scheduler is one of those things people warn you away from >> though. I wish there was a solid library we could embed, but I haven't >> found anything. >> >> I wanted to get some opinions from this community. Do you know of any >> scheduling libraries? Do you think having scheduling built in is a good >> idea? A bad idea? Should we keep chipping away at it? Would people be >> interested in a standalone scheduling library, or is this problem >> inherently too complex to be adequately captured in library form? >> >> Thanks! >> >> Funnel: https://github.com/ohsu-comp-bio/funnel >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
