Hi, I've been trying to discuss this idea here as you suggested on stackoverflow but never had any respone in more than 10 days... I guess you must be busy with other things at the moment. So I went along and forked the source code to make the necessary changes and make my solution work. I pushed it there a week ago:
http://michakfromparis.github.io/gnu-parallel-sticky/ the source code is available on github here: https://github.com/michaKFromParis/gnu-parallel-sticky Wasn't exactly easy without any guidance as the source code has a lot of history so I tried to keep the changes surgical to ease merge of your future releases. I've been using it in production for more than a week now and it works perfectly in my configuration. It is also compatible with older formats, should be a drop-in replacement for usual parallel uses with extra features on the side. Would love to get feedback from other users though as it might not be completely dry. Thanks for sharing the original source code. Best Regards, M On Fri, Oct 31, 2014 at 9:18 AM, Michel Courtine <mic...@ivoltage.me> wrote: > Hi again, > > I started digging into the code and managed to add support for SSHLogin > tags and Arg tags pretty cleanly. > I add to change the syntax as you were already splitting on the ',' for > the sshlogin strings and my new syntax looks like this in a simplified way: > > ./parallel -j1 --verbose --tag --sshlogin 4/*iOS+Android*/silver > --sshlogin 8/*OSX*/red "echo" ::: *ir@iOS* *ar@Android or@OSX* > > When it comes to jobs submission, it gets a bit more complicated as you > visibly iterate on the sshlogins and only then parse the jobs args one by > one. Am I correct? > I managed to insert a "tag matching" test into get_job_with_sshlogin but > returning undef aborts the job forever obviously. > What I would really need is reverse the loop and iterate on the jobs to > look for a matching host but I don't want to restructure the code so deeply. > I feel a bit stuck there. Do you have any idea of an elegant way to reach > my goal? > Do you have a dev branch I could push to for you to review my changes? > > Best, > > M > > On Thu, Oct 30, 2014 at 7:50 PM, Michel Courtine <mic...@ivoltage.me> > wrote: > >> Hi, >> >> Following-up from >> http://stackoverflow.com/questions/26645042/gnu-parallel-host-sticky-jobs >> >> I truly believe this is an important feature to implement as it is quite >> simple, completely generic and seems to be an obvious need in many user >> scenarios to me. >> >> I finally realized parallel is written in perl and all the source code >> stands in one file. Silly me. I was expecting some c code. >> I must admit my perl is quite rusty, haven't used it much since >> university but I could dig into it a bit if you give me some guidance. >> I had a very quick look at the code and it looks like the host list is >> held inside Global::host so it shouldn't be too hard to add a list of tags >> to it. But we still need to define a cli syntax for it. >> I quite like the 4/sshstring syntax. What about adding a second optional >> slash with values separated by comas? >> >> >> *Old fashion, of course still compatible* >> >> hostname >> user@hostname >> 4/hostname >> >> *Optional new fashion* >> >> *4/iOS,Android/hostname1* >> *8/Linux/hostname2* >> *8/OSX/hostname3* >> *16/Emscripten/hostname4* >> >> I am less sure about the right way to tag the jobs though. Use the same >> syntax? >> >> *iOS/ir* >> *Android/ar* >> *Linux/lr* >> *OSX/or* >> *Emscripten/er* >> >> This would invalidate the '/' in the input sources though. Not an issue >> for me but could probably be an issue for other users. >> >> parallel -j 1 --verbose --tag --filter-host \ >> --basefile ./ci-builder --basefile ./utils --return {} --result >> ./logs/ \ >> --sshlogin *4/iOS,Android/hostname1 \* >> --sshlogin *8/Linux/hostname2** \* >> --sshlogin *8/OSX/hostname3** \* >> --sshlogin *16/Emscripten/hostname4** \* >> "TERM=xterm;SHELL=/bin/bash;PATH=/usr/local/bin:$PATH;export >> BUILD_PROJECT_NAME=$BUILD_PROJECT_NAME;export >> BUILD_PROJECT_URL=$BUILD_PROJECT_URL;./ci-builder" ::: \ >> * iOS/ir \* >> * Android/ar \* >> * Linux/lr \* >> * OSX/or \* >> * Emscripten/er* >> >> *I would then expect* >> >> ir to be sent to hostname1 >> ar to be put on hold for ir to complete >> lr to be sent to hostname2 >> or to be sent to hostname3 >> er to be sent to hostname4 >> >> *Optional Behaviour* >> >> One important option that seems to make sense is a 'strict-stickyness' >> which is the behavior described above >> i.e. if a job requires a tagged host and all these tagged nodes are busy, >> the job is put on hold, waiting for the one of the tagged hosts to free up. >> It would be nice to be able to disable the strict aspect of stickyness so >> a job would then go to the next available node regardless of tags. >> >> *Error Handling* >> >> The only new kind of errors that is introduced is a mismatched label >> defined on the jobs that is not defined on the hosts. I think the script >> should simply abort at launch with a nice error message >> >> >> *Additional point* >> >> At the moment, the jobs seem to be distributed in a random manner, I >> think it would be nice to add an option to make the job submission order >> stick to the sshlogin order >> >> What do you think? >> >> Best Regards, >> >> M >> >> -- >> Michel Courtine | CEO | iVoltage >> +33 6 189 354 89 | mic...@ivoltage.me >> http://ivoltage.me >> > > > > -- > Michel Courtine | CEO | iVoltage > +33 6 189 354 89 | mic...@ivoltage.me > http://ivoltage.me > -- Michel Courtine | CEO | iVoltage +33 6 189 354 89 | mic...@ivoltage.me http://ivoltage.me
