Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Lior Okman
Hi, You could try something like Concourse ( https://concourse-ci.org/ ). It allows you to define a pipeline which is comprised of jobs and the order in which they should be invoked. -- Lior On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe wrote: > Hi all, > > I need some advice,

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Steve Litt
On Tue, 19 Jun 2018 09:42:35 +0300 Omer Zak wrote: > For dependency management, you may want to use 'make' or modern Hi Omer, While corresponding with someone offlist, I had another idea maybe as good as using make. I could make a customized installation of the process supervisor part of

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Steve Litt
On Tue, 19 Jun 2018 12:25:21 +0300 Dimid Duchovny wrote: > Hi Rabin, > > > I'm far from being a linux expert, but isn't dependency between > services handled by systemd? > E.g. https://wiki.archlinux.org/index.php/Systemd/Timers If you drive on that side of the road :-) More seriously, I

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Steve Litt
On Tue, 19 Jun 2018 09:42:35 +0300 Omer Zak wrote: > For dependency management, you may want to use 'make' If you can depend on each task to create specific files, yeah, that sounds like a great idea. I should have thought of it. And then you just put it in a loop so things are always being

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread linux.il
I suggest to check Jenkins (as already suggested) and Rundeck. Vitaly On Tue, Jun 19, 2018 at 9:06 AM, Rabin Yasharzadehe wrote: > Hi all, > > I need some advice, currently I have a huge cron file which schedules > tasks one after anther, and each task is position precisely (with some room >

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Moish
try GNUbatch On 19/06/2018 09:06, Rabin Yasharzadehe wrote: Hi all, I need some advice, currently I have a huge cron file which schedules tasks one after anther, and each task is

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Omer Zak
1. Execution time limits: Ansible has async with polling intervals. I did not research for methods to kill hung tasks. https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html 2. Dashboard-like functionality According to:

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Ari Becker
Hi Rabin, Did you consider using Jenkins? It may be a little heavyweight, but it should be relatively easy to set up and configure. You can use the same scripts you're using today, the ability to state which jobs run on which nodes, set up dependencies between them, set timeouts, set cron

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Rabin Yasharzadehe
systemd is a complete different tool, which was not designed for this kinda purpose. (maybe in the future it will grow to be something like that ;-) ) I'm looking for something a bit more sophisticated then "go to this machine" and "run this script" and "expect this result" i like to define

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Dimid Duchovny
Hi Rabin, I'm far from being a linux expert, but isn't dependency between services handled by systemd? E.g. https://wiki.archlinux.org/index.php/Systemd/Timers HTH 2018-06-19 12:12 GMT+03:00 Moish : > Try GNUbatch. > > > On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak wrote: > >> For

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Moish
Try GNUbatch. On June 19, 2018 9:42:35 AM GMT+03:00, Omer Zak wrote: >For dependency management, you may want to use 'make' or modern >equivalents ('ant', 'gradle', etc.). >For controlling remote nodes, 'ansible' may be able to do the work. > >--- Omer Zak > > >On Tue, 2018-06-19 at 09:06

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Rabin Yasharzadehe
I'll have to read the documentation to learn more, but this project seems barely maintained as only minor versions each year or two (last release was 2 years ago), that doesn't give a lot of confidence. but i'll check it out thanks. -- Rabin On Tue, 19 Jun 2018 at 09:43, Marc Volovic wrote:

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Omer Zak
For dependency management, you may want to use 'make' or modern equivalents ('ant', 'gradle', etc.). For controlling remote nodes, 'ansible' may be able to do the work. --- Omer Zak On Tue, 2018-06-19 at 09:06 +0300, Rabin Yasharzadehe wrote: > Hi all,  > > I need some advice, currently I have

Re: Moving away from cron jobs to some workflow manager

2018-06-19 Thread Rabin Yasharzadehe
never heard of it, but from reading the manual and the 10minute presentation , it's seems like it is more suitable for data crunching, where you have a pool of compute resources and you submit jobs to it. my case is a bit different, where I have many jobs which need to run (orchestrated) on there

Moving away from cron jobs to some workflow manager

2018-06-19 Thread Rabin Yasharzadehe
Hi all, I need some advice, currently I have a huge cron file which schedules tasks one after anther, and each task is position precisely (with some room for error) to start after it predecessor. So if one job start at 00:00 and it will go and fetch some files and it takes 3minutes the next job