Hi,
Following-up from
http://stackoverflow.com/questions/26645042/gnu-parallel-host-sticky-jobs
I truly believe this is an important feature to implement as it is quite
simple, completely generic and seems to be an obvious need in many user
scenarios to me.
I finally realized parallel is written in perl and all the source code
stands in one file. Silly me. I was expecting some c code.
I must admit my perl is quite rusty, haven't used it much since university
but I could dig into it a bit if you give me some guidance.
I had a very quick look at the code and it looks like the host list is held
inside Global::host so it shouldn't be too hard to add a list of tags to
it. But we still need to define a cli syntax for it.
I quite like the 4/sshstring syntax. What about adding a second optional
slash with values separated by comas?
*Old fashion, of course still compatible*
hostname
user@hostname
4/hostname
*Optional new fashion*
*4/iOS,Android/hostname1*
*8/Linux/hostname2*
*8/OSX/hostname3*
*16/Emscripten/hostname4*
I am less sure about the right way to tag the jobs though. Use the same
syntax?
*iOS/ir*
*Android/ar*
*Linux/lr*
*OSX/or*
*Emscripten/er*
This would invalidate the '/' in the input sources though. Not an issue for
me but could probably be an issue for other users.
parallel -j 1 --verbose --tag --filter-host \
--basefile ./ci-builder --basefile ./utils --return {} --result ./logs/
\
--sshlogin *4/iOS,Android/hostname1 \*
--sshlogin *8/Linux/hostname2** \*
--sshlogin *8/OSX/hostname3** \*
--sshlogin *16/Emscripten/hostname4** \*
"TERM=xterm;SHELL=/bin/bash;PATH=/usr/local/bin:$PATH;export
BUILD_PROJECT_NAME=$BUILD_PROJECT_NAME;export
BUILD_PROJECT_URL=$BUILD_PROJECT_URL;./ci-builder" ::: \
* iOS/ir \*
* Android/ar \*
* Linux/lr \*
* OSX/or \*
* Emscripten/er*
*I would then expect*
ir to be sent to hostname1
ar to be put on hold for ir to complete
lr to be sent to hostname2
or to be sent to hostname3
er to be sent to hostname4
*Optional Behaviour*
One important option that seems to make sense is a 'strict-stickyness'
which is the behavior described above
i.e. if a job requires a tagged host and all these tagged nodes are busy,
the job is put on hold, waiting for the one of the tagged hosts to free up.
It would be nice to be able to disable the strict aspect of stickyness so a
job would then go to the next available node regardless of tags.
*Error Handling*
The only new kind of errors that is introduced is a mismatched label
defined on the jobs that is not defined on the hosts. I think the script
should simply abort at launch with a nice error message
*Additional point*
At the moment, the jobs seem to be distributed in a random manner, I think
it would be nice to add an option to make the job submission order stick to
the sshlogin order
What do you think?
Best Regards,
M
--
Michel Courtine | CEO | iVoltage
+33 6 189 354 89 | mic...@ivoltage.me
http://ivoltage.me
