Re: [Toolserver-l] Cronjob starting cronsub
On Tue, Mar 1, 2011 at 12:49 AM, MZMcBride z...@mzmcbride.com wrote: Magnus Manske wrote: I understand that, and I didn't say lfs was better (it has plenty of its own quirks), or that SGE was a bad choice - it seems to run well enough, all things considered. It's just that things like SGE might interpret comments in your scripts as commands and die or you can't run qsub from within qsub are types of behaviour that are unusual to say the least, and the toolserver users are left to the mercy of these issues (I'd call them bugs) without warning or documentation. (Yes, there is a wiki pages that says it can interpret special comments in your script. That's somewhat different from it will reinterpret your Perl script and fall over). Having learned some lessons today, you should write improved documentation on the wiki. :-) I might, once I've recovered from said learning experience... Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Sun, Feb 27, 2011 at 1:10 PM, River Tarnell r.tarn...@ieee.org wrote: In article slrnimkfjp.fte.sa...@saper.info, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: I have a wrapper script that starts a handful of jobs on the toolserver (via qsub). But, when I put that wrapper script as a normal cronjob, I get some error messages about qsub not found. Starting the wrapper via cronsub doesn't work either. You should first run cronsub from the command-line with the same arguments. If that still doesn't work, post the entire command line and any output from it. Once you have it working from the command-line then it should be fine fromc ron. Actually cronsub does a litle bit more than sourcing /sge62/default/common/settings.sh file, so I don't know why it's not working for you. Maybe you could share some output? cronsub makes it easier to submit longrun jobs, automatically sets the default output file, runs the script from $HOME and prevents duplicate jobs being submitted. Otherwise it's identical to qsub. OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. Script ( /home/magnus/cronjobs/most_wanted_wrapper.sh ) : #!/bin/ksh qsub -N mw_dewiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.dewiki_p.out -l sqlprocs-s5=1 /home/magnus/cronjobs/update_most_wanted.pl dewiki_p qsub -N mw_enwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.enwiki_p.out -l sqlprocs-s1=1 /home/magnus/cronjobs/update_most_wanted.pl enwiki_p qsub -N mw_frwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.frwiki_p.out -l sqlprocs-s6=1 /home/magnus/cronjobs/update_most_wanted.pl frwiki_p qsub -N mw_plwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.plwiki_p.out -l sqlprocs-s2=1 /home/magnus/cronjobs/update_most_wanted.pl plwiki_p Running this script, or any of its individual lines, directly from shell, works absolutely fine. However, on cronsub mostwanted /home/magnus/cronjobs/most_wanted_wrapper.sh I get (in the error file): /home/magnus/cronjobs/most_wanted_wrapper.sh[2]: 7274 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[3]: 7289 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[4]: 7290 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[5]: 7298 Segmentation Fault(coredump) I tried replacing qsub with cronsub, only to be met with a flood of errors relating to all the options it doesn't know, including the sql slot reservation (I guess that's my punishment for trying to be a good citizen...). Removing basically all options, using only lines like this: cronsub -s mw_dewiki_p /home/magnus/cronjobs/update_most_wanted.pl dewiki_p in the wrapper script, again runs peachy from shell, but barfs coredumps again when run via cronsub. So, as far as I am concerned, it's broken. Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
In article aanlktiknfjfxqdvgzcj63hz90v_xpmkqvu145yj+p...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. This should not be the case. Please provide the exact script and command line you used and the error message. - river. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell r.tarn...@ieee.org wrote: In article aanlktiknfjfxqdvgzcj63hz90v_xpmkqvu145yj+p...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. This should not be the case. Please provide the exact script and command line you used and the error message. cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 1:59 PM, River Tarnell r.tarn...@ieee.org wrote: In article AANLkTi=cdpcjx_znz04wxmqiw6ssvz3nphywu7-oj...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option What is the output from this? $ ksh -x /opt/local/bin/cronsub update_dupes /home/magnus/update_dupes.pl + . /sge62/default/common/settings.sh + SGE_ROOT=/sge62 + export SGE_ROOT + + /sge62/util/arch ARCH=sol-amd64 + + /sge62/util/arch -m DEFAULTMANPATH=/usr/share/man + + /sge62/util/arch -mt MANTYPE=man + SGE_CELL=default + export SGE_CELL + SGE_CLUSTER_NAME=toolserver + export SGE_CLUSTER_NAME + unset SGE_QMASTER_PORT + unset SGE_EXECD_PORT + [ /sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man = ] + MANPATH=/sge62/man:/sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man + export MANPATH + PATH=/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + export PATH + unset ARCH DEFAULTMANPATH MANTYPE + PATH=/usr/bin:/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + sflag= + QUEUE=all.q + OPTS= + hrt= + getopts slh f + shift 0 + [[ 2 -ge 1 ]] + JOBNAME=update_dupes + shift + [ -z ] + [[ 1 -ge 1 ]] + SCRIPT=/home/magnus/update_dupes.pl + + mktemp TMPF=/tmp/tmp.5jaiw2 + chmod 700 /tmp/tmp.5jaiw2 + trap rm -f /tmp/tmp.5jaiw2 0 + cat + 1 /tmp/tmp.5jaiw2 0 #! /bin/sh cd $HOME exec $@ exit 1 + grep ^#\$ + 0 /home/magnus/update_dupes.pl 1 /tmp/tmp.5jaiw2 + grep Full jobname:[ \t]*update_dupes$ + 1 /dev/null + qstat -r + qsub -q all.q -N update_dupes /tmp/tmp.5jaiw2 /home/magnus/update_dupes.pl + 1 /dev/null qsub: Unknown option qsub: Unknown option qsub: Unknown option + rm -f /tmp/tmp.5jaiw2 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell r.tarn...@ieee.org wrote: In article aanlktiknfjfxqdvgzcj63hz90v_xpmkqvu145yj+p...@mail.gmail.com, Magnus Manske magnusman...@googlemail.com wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. This should not be the case. Please provide the exact script and command line you used and the error message. cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option This is because you have in your script #$cmd = ./cron_sql.sh unless -e $cmd ; and this is recognized by qsub(1) as the embedded qsub option. If you change it to: # $cmd = it works again. Thanks! Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo meaning from my scripts (which predate qsub availability) anyway? If that is desired for some cases, why is that activated by default? Is there at least a qsub option to turn it off? Will that option work with cronsub? In my day job, I work with a large compute farm (lfs, ~10K cores), so I'm more than happy to work with, and migrate existing cronjobs to, this system, but it would help if it weren't a deliberate minefield... Cheers, Magnus ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
Magnus Manske magnusman...@googlemail.com wrote: On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: cronsub update_dupes /home/magnus/update_dupes.pl qsub: Unknown option qsub: Unknown option qsub: Unknown option This is because you have in your script #$cmd = ./cron_sql.sh unless -e $cmd ; and this is recognized by qsub(1) as the embedded qsub option. If you change it to: # $cmd = it works again. Thanks! Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo meaning from my scripts (which predate qsub availability) anyway? If that is desired for some cases, why is that activated by default? Is there at least a qsub option to turn it off? Will that option work with cronsub? To quote man qsub(1): -C prefix_string Available for qsub and qrsh with script submission (-b n). Prefix_string defines the prefix that declares a direc- tive in the job's command. The prefix is not a job attribute, but affects the behavior of qsub and qrsh. If prefix is a null string, the command will not be scanned for embedded directives. The directive prefix consists of two ASCII characters which, when appearing in the first two bytes of a script line, indicate that what follows is an Sun Grid Engine command. The default is #$. The user should be aware that changing the first delim- iting character can produce unforeseen side effects. If the script file contains anything other than a # character in the first byte position of the line, the shell processor for the job will reject the line and may exit the job prematurely. If the -C option is present in the script file, it is ignored. However, please be warned that cronsub currently messes up custom qsub options. //Marcin ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
On Mon, Feb 28, 2011 at 3:01 PM, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: On Mon, Feb 28, 2011 at 11:43 AM, Ilmari Karonen nos...@vyznev.net wrote: On 02/28/2011 01:00 PM, Magnus Manske wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?) Any parameter? I smell a line break encoding issue. Try making sure there are no carriage returns in your crontab or in any of the scripts involved. Well, since that happens also when I start cronsub fron the shell, I doubt it... It seems like qsub dumps core when executed from inside of the SGE job. /sge62/default/spool/wolfsbane/job_scripts/210756[3]: 1253 Segmentation Fault(coredump) I have filed https://jira.toolserver.org/browse/TS-963 about this. Really strange. Maybe some safety mechanism to prevent infinite recursion gone bad? ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Cronjob starting cronsub
In article slrnimkfjp.fte.sa...@saper.info, Marcin Cieslak sa...@saper.info wrote: Magnus Manske magnusman...@googlemail.com wrote: I have a wrapper script that starts a handful of jobs on the toolserver (via qsub). But, when I put that wrapper script as a normal cronjob, I get some error messages about qsub not found. Starting the wrapper via cronsub doesn't work either. You should first run cronsub from the command-line with the same arguments. If that still doesn't work, post the entire command line and any output from it. Once you have it working from the command-line then it should be fine fromc ron. Actually cronsub does a litle bit more than sourcing /sge62/default/common/settings.sh file, so I don't know why it's not working for you. Maybe you could share some output? cronsub makes it easier to submit longrun jobs, automatically sets the default output file, runs the script from $HOME and prevents duplicate jobs being submitted. Otherwise it's identical to qsub. - river. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
