[Wikitech-l] Re: How to get alerts on errors for cron jobs on toolforge
Cron should send mail with the output of the command in the crontab. Because the grid crontab runs jsub, and not your command directly, the only mail you will get from cron is when your job has been submitted. You can use the -m option on jsub to send mail in a few different cases (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#jsub_options). jsub -m a ... should send mail when the job is aborted, but won't include any more information. You can also send mail directly from your code, https://wikitech.wikimedia.org/wiki/Help:Toolforge/Email has some more information. Many logging frameworks, including `logging` in Python, support sending log messages over SMTP. ACN On Wed, Jun 23, 2021 at 5:03 AM QEDK wrote: > > I do remember getting emails (last year?) from running cronjobs on the grid > until I disabled them, iirc it was when they failed, but not necessarily a > description of the error. I literally cannot find any documentation regarding > this anymore so I almost have to think if I didn't imagine this entire ordeal. > > Best, > QEDK > > On Wed, Jun 23, 2021, 09:01 Sam Wilson wrote: >> >> I think the only way is to have your cronjob do the emailing of the >> output, and then delete (or move) the output file so it's not repeated >> on the next run. Something like: >> >> #!/bin/bash >> >> ## Call actual job script here. >> ./do-tool-things.sh >> >> ## Email output to maintainers. >> echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain; >> charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out | >> /usr/sbin/exim -odf -i tools.tooln...@tools.wmflabs.org >> >> ## Remove output file. >> rm /data/project/toolname/cron-tools.jobname.out >> >> Although of course the details might differ for different tools. >> >> >> On 22/6/21 4:52 pm, Shrinivasan T wrote: >> > Hello all, >> > >> > I have a bot at toolforge, running weekly once in cron job using "jsub" >> > >> > Followed the below link for the setup. >> > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot >> > >> > Getting the JOBNAME.out and JOBNAME.err files at the home folder. >> > >> > Is there any other way like email notification to get alerts, if some >> > error happens? >> > >> > I think the files JOBNAME.out and JOBNAME.err will be keep on growing >> > on the bason host login.toolforge.org >> > >> > How to avoid this? >> > >> > Thanks. >> > >> ___ >> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org >> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org >> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ > > ___ > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ ___ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
[Wikitech-l] Re: How to get alerts on errors for cron jobs on toolforge
I do remember getting emails (last year?) from running cronjobs on the grid until I disabled them, iirc it was when they failed, but not necessarily a description of the error. I literally cannot find any documentation regarding this anymore so I almost have to think if I didn't imagine this entire ordeal. Best, QEDK On Wed, Jun 23, 2021, 09:01 Sam Wilson wrote: > I think the only way is to have your cronjob do the emailing of the > output, and then delete (or move) the output file so it's not repeated > on the next run. Something like: > > #!/bin/bash > > ## Call actual job script here. > ./do-tool-things.sh > > ## Email output to maintainers. > echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain; > charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out | > /usr/sbin/exim -odf -i tools.tooln...@tools.wmflabs.org > > ## Remove output file. > rm /data/project/toolname/cron-tools.jobname.out > > Although of course the details might differ for different tools. > > > On 22/6/21 4:52 pm, Shrinivasan T wrote: > > Hello all, > > > > I have a bot at toolforge, running weekly once in cron job using "jsub" > > > > Followed the below link for the setup. > > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot > > > > Getting the JOBNAME.out and JOBNAME.err files at the home folder. > > > > Is there any other way like email notification to get alerts, if some > > error happens? > > > > I think the files JOBNAME.out and JOBNAME.err will be keep on growing > > on the bason host login.toolforge.org > > > > How to avoid this? > > > > Thanks. > > > ___ > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ > ___ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
[Wikitech-l] Re: How to get alerts on errors for cron jobs on toolforge
I think the only way is to have your cronjob do the emailing of the output, and then delete (or move) the output file so it's not repeated on the next run. Something like: #!/bin/bash ## Call actual job script here. ./do-tool-things.sh ## Email output to maintainers. echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain; charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out | /usr/sbin/exim -odf -i tools.tooln...@tools.wmflabs.org ## Remove output file. rm /data/project/toolname/cron-tools.jobname.out Although of course the details might differ for different tools. On 22/6/21 4:52 pm, Shrinivasan T wrote: Hello all, I have a bot at toolforge, running weekly once in cron job using "jsub" Followed the below link for the setup. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot Getting the JOBNAME.out and JOBNAME.err files at the home folder. Is there any other way like email notification to get alerts, if some error happens? I think the files JOBNAME.out and JOBNAME.err will be keep on growing on the bason host login.toolforge.org How to avoid this? Thanks. ___ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
