Hi again, I started digging into the code and managed to add support for SSHLogin tags and Arg tags pretty cleanly. I add to change the syntax as you were already splitting on the ',' for the sshlogin strings and my new syntax looks like this in a simplified way:
./parallel -j1 --verbose --tag --sshlogin 4/*iOS+Android*/silver --sshlogin 8/*OSX*/red "echo" ::: *ir@iOS* *ar@Android or@OSX* When it comes to jobs submission, it gets a bit more complicated as you visibly iterate on the sshlogins and only then parse the jobs args one by one. Am I correct? I managed to insert a "tag matching" test into get_job_with_sshlogin but returning undef aborts the job forever obviously. What I would really need is reverse the loop and iterate on the jobs to look for a matching host but I don't want to restructure the code so deeply. I feel a bit stuck there. Do you have any idea of an elegant way to reach my goal? Do you have a dev branch I could push to for you to review my changes? Best, M On Thu, Oct 30, 2014 at 7:50 PM, Michel Courtine <mic...@ivoltage.me> wrote: > Hi, > > Following-up from > http://stackoverflow.com/questions/26645042/gnu-parallel-host-sticky-jobs > > I truly believe this is an important feature to implement as it is quite > simple, completely generic and seems to be an obvious need in many user > scenarios to me. > > I finally realized parallel is written in perl and all the source code > stands in one file. Silly me. I was expecting some c code. > I must admit my perl is quite rusty, haven't used it much since university > but I could dig into it a bit if you give me some guidance. > I had a very quick look at the code and it looks like the host list is > held inside Global::host so it shouldn't be too hard to add a list of tags > to it. But we still need to define a cli syntax for it. > I quite like the 4/sshstring syntax. What about adding a second optional > slash with values separated by comas? > > > *Old fashion, of course still compatible* > > hostname > user@hostname > 4/hostname > > *Optional new fashion* > > *4/iOS,Android/hostname1* > *8/Linux/hostname2* > *8/OSX/hostname3* > *16/Emscripten/hostname4* > > I am less sure about the right way to tag the jobs though. Use the same > syntax? > > *iOS/ir* > *Android/ar* > *Linux/lr* > *OSX/or* > *Emscripten/er* > > This would invalidate the '/' in the input sources though. Not an issue > for me but could probably be an issue for other users. > > parallel -j 1 --verbose --tag --filter-host \ > --basefile ./ci-builder --basefile ./utils --return {} --result > ./logs/ \ > --sshlogin *4/iOS,Android/hostname1 \* > --sshlogin *8/Linux/hostname2** \* > --sshlogin *8/OSX/hostname3** \* > --sshlogin *16/Emscripten/hostname4** \* > "TERM=xterm;SHELL=/bin/bash;PATH=/usr/local/bin:$PATH;export > BUILD_PROJECT_NAME=$BUILD_PROJECT_NAME;export > BUILD_PROJECT_URL=$BUILD_PROJECT_URL;./ci-builder" ::: \ > * iOS/ir \* > * Android/ar \* > * Linux/lr \* > * OSX/or \* > * Emscripten/er* > > *I would then expect* > > ir to be sent to hostname1 > ar to be put on hold for ir to complete > lr to be sent to hostname2 > or to be sent to hostname3 > er to be sent to hostname4 > > *Optional Behaviour* > > One important option that seems to make sense is a 'strict-stickyness' > which is the behavior described above > i.e. if a job requires a tagged host and all these tagged nodes are busy, > the job is put on hold, waiting for the one of the tagged hosts to free up. > It would be nice to be able to disable the strict aspect of stickyness so > a job would then go to the next available node regardless of tags. > > *Error Handling* > > The only new kind of errors that is introduced is a mismatched label > defined on the jobs that is not defined on the hosts. I think the script > should simply abort at launch with a nice error message > > > *Additional point* > > At the moment, the jobs seem to be distributed in a random manner, I think > it would be nice to add an option to make the job submission order stick to > the sshlogin order > > What do you think? > > Best Regards, > > M > > -- > Michel Courtine | CEO | iVoltage > +33 6 189 354 89 | mic...@ivoltage.me > http://ivoltage.me > -- Michel Courtine | CEO | iVoltage +33 6 189 354 89 | mic...@ivoltage.me http://ivoltage.me
