> > > Having tried using things like gleam/glow (see > https://github.com/chrislusf/gleam) and dask (which has similar ideas but > written in python) it sounds like a FIFO work stealing type scheduler > wouldn't be a bad idea. In particular dask's approach is quite nicely put > together. Though one thing that I have noticed with a lot of these non-HPC > type schedulers is that they are great for doing embarrassingly parallel > type problems but fall apart reasonably quickly once you get into MPI > territory where you want to be careful about how you layout the > processes/network for your job, probably things to consider when writing a > scheduler. >
Ya, Funnel currently has a really simple scheduler along these lines, but the minute it had two users, we realized we wanted something smarter (fair scheduling). Hadn't considered MPI yet, as I'm more interested in large batch jobs, but I can see people wanting it, and data locality, and worker admin, and, and, and pretty soon you've just built Kubernetes :) Many projects have done this though, so it seems like a well understood problem, so it seems like something that could stand on its own as a library, built in Go and wrapped by other languages via C bindings. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
