I'll have to read the documentation to learn more, but this project seems barely maintained as only minor versions each year or two (last release was 2 years ago), that doesn't give a lot of confidence.
but i'll check it out thanks. -- Rabin On Tue, 19 Jun 2018 at 09:43, Marc Volovic <marcvolo...@icloud.com> wrote: > Hi, > > It is intended for submitting multiple jobs for crunching. But you can use > it (SOGE) or SLURM for issuing job and dependent jobs, even on a single > machine issuer/execution host. It can be used as a resource aware job > scheduler. > > —mav > Marc Volovic > marcvolo...@me.com > > > > > On 19 Jun 2018, at 9:41, Rabin Yasharzadehe <ra...@rabin.io> wrote: > > > > never heard of it, > > but from reading the manual and the 10minute presentation , > > it's seems like it is more suitable for data crunching, where you have a > pool > > of compute resources and you submit jobs to it. > > > > my case is a bit different, where I have many jobs which need to run > (orchestrated) on there own hosts > > with a specific environment and setup. > > > > > > -- > > Rabin > > > > > > On Tue, 19 Jun 2018 at 09:10, Marc Volovic <marcvolo...@icloud.com> > wrote: > > Why not a minimal deploy of SGE - which would also allow you to make > multi-executor? > > > > https://arc.liv.ac.uk/trac/SGE > > > > —mav > > Marc Volovic > > marcvolo...@me.com > > > > > > > > > On 19 Jun 2018, at 9:06, Rabin Yasharzadehe <ra...@rabin.io> wrote: > > > > > > Hi all, > > > > > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room > for error) to start after it predecessor. > > > > > > So if one job start at 00:00 and it will go and fetch some files and > it takes 3minutes > > > the next job will be after start right after at ~00:05 > > > and so on > > > > > > the problem is that if one job fails, all other jobs which are depend > on him will fail as well, and then I get a shitload of alerts, and the > worst part is that if i have to manually start a batch process I need to go > to each machine and manually start each job in the right order, > > > > > > I was looking to resolve this problem with a tool which can manage > this "pipe line" > > > and I cam across several tools like Luigi and (apache-)AirFlow, I > started with Luigi but It didn't look > > > right for the job, and then I tried airflow, but was not able to make > it to work, the jobs queue never executed. =( > > > > > > Has any one have experience with airflow, or other tool like it which > they can recommend ? > > > My needs are to be able to execute my CURRENT shell/python/php scripts > and build the dependency between them, and I perfer the option for remote > exec so that I will have central > > > place to manage and monitor all work flow whichs are executed on > several nodes. > > > > > > Thanks in advance, > > > Rabin > > > _______________________________________________ > > > Linux-il mailing list > > > Linux-il@cs.huji.ac.il > > > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > > >
_______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il