Hi, What is this fab ? Python scripts with shebang (self executable)? python -u usually helps with jenkins python output issues.
On Wed, Sep 17, 2014 at 11:38 PM, Michael Reinhardt <[email protected]> wrote: > We have a Python fabric command that runs in parallel across several > hosts, something like this: > > $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel > cache_copy > > This will copy cache to the production servers listed in parallel. There > is various logging that occurs throughout the process to indicate how far > along we are since it can take hours for the XXgig cache directories. Since > the copying is happening concurrently, the output when run on the command > line comes back interlaced in real-time, like so: > > [prod1.server] Executing task 'cache_copy' > [prod2.server] Executing task 'cache_copy' > [prod3.server] Executing task 'cache_copy' > 2014-09-16 10:02:29.688243 > [prod1.server] INFO: rsyncing cache dir > [prod1.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:29.716345 > [prod2.server] INFO: rsyncing cache dir > [prod2.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:29.853275 > [prod3.server] INFO: rsyncing cache dir > [prod3.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:29.984154 > [prod1.server] INFO: Reloading nginx config > [prod1.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.025155 > [prod2.server] INFO: Reloading nginx config > [prod2.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.100169 > [prod1.server] SUCCESS: CACHE COPY COMPLETE > > 2014-09-16 10:02:30.181938 > [prod2.server] SUCCESS: CACHE COPY COMPLETE > > 2014-09-16 10:02:30.331402 > [prod3.server] INFO: Reloading nginx config > [prod3.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.559271 > [prod3.server] SUCCESS: CACHE COPY COMPLETE > > Done. > > However, when the task is run through Jenkins, the console output does not > display until all tasks are done because Jenkins groups the output AFTER > the threads are joined when all threads are complete. So, once all commands > are complete the output looks like this: > > [prod1.server] Executing task 'cache_copy' > 2014-09-16 10:02:29.688243 > [prod1.server] INFO: rsyncing cache dir > [prod1.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:29.984154 > [prod1.server] INFO: Reloading nginx config > [prod1.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.100169 > [prod1.server] SUCCESS: CACHE COPY COMPLETE > > [prod2.server] Executing task 'cache_copy' > 2014-09-16 10:02:29.716345 > [prod2.server] INFO: rsyncing cache dir > [prod2.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:30.025155 > [prod2.server] INFO: Reloading nginx config > [prod2.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.181938 > [prod2.server] SUCCESS: CACHE COPY COMPLETE > > [prod3.server] Executing task 'cache_copy' > 2014-09-16 10:02:29.853275 > [prod3.server] INFO: rsyncing cache dir > [prod3.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" > \ > "repo/cache/." > 2014-09-16 10:02:30.331402 > [prod3.server] INFO: Reloading nginx config > [prod3.server] run: sbin/nginx -s reload -c "repo/nginx.conf" > 2014-09-16 10:02:30.559271 > [prod3.server] SUCCESS: CACHE COPY COMPLETE > > Done. > > While this is more readable, it's not ideal because we'd like to keep > track of the state of the process by reading the console output in > realtime. Note that when this fabric command is run *without* the > `--parallel` option the console output *does occur in realtime*, however > obviously this is not workable because the serial process takes much longer > to run. > > I haven't been able to find a setting in Jenkins that will disable this > thread grouping. Does anyone have any ideas? > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
