[Wikitech-l] Re: How to get alerts on errors for cron jobs on toolforge

[AntiCompositeNumber]

Cron should send mail with the output of the command in the crontab.
Because the grid crontab runs jsub, and not your command directly, the
only mail you will get from cron is when your job has been submitted.

You can use the -m option on jsub to send mail in a few different
cases (see 
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#jsub_options).
jsub -m a ... should send mail when the job is aborted, but won't
include any more information.

You can also send mail directly from your code,
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Email has some more
information. Many logging frameworks, including `logging` in Python,
support sending log messages over SMTP.

ACN

On Wed, Jun 23, 2021 at 5:03 AM QEDK  wrote:
>
> I do remember getting emails (last year?) from running cronjobs on the grid 
> until I disabled them, iirc it was when they failed, but not necessarily a 
> description of the error. I literally cannot find any documentation regarding 
> this anymore so I almost have to think if I didn't imagine this entire ordeal.
>
> Best,
> QEDK
>
> On Wed, Jun 23, 2021, 09:01 Sam Wilson  wrote:
>>
>> I think the only way is to have your cronjob do the emailing of the
>> output, and then delete (or move) the output file so it's not repeated
>> on the next run. Something like:
>>
>> #!/bin/bash
>>
>> ## Call actual job script here.
>> ./do-tool-things.sh
>>
>> ## Email output to maintainers.
>> echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain;
>> charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out |
>> /usr/sbin/exim -odf -i tools.tooln...@tools.wmflabs.org
>>
>> ## Remove output file.
>> rm /data/project/toolname/cron-tools.jobname.out
>>
>> Although of course the details might differ for different tools.
>>
>>
>> On 22/6/21 4:52 pm, Shrinivasan T wrote:
>> > Hello all,
>> >
>> > I have a bot at toolforge, running weekly once in cron job using "jsub"
>> >
>> > Followed the below link for the setup.
>> > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
>> >
>> > Getting the JOBNAME.out and JOBNAME.err files at the home folder.
>> >
>> > Is there any other way like email notification to get alerts, if some
>> > error happens?
>> >
>> > I think the files JOBNAME.out and JOBNAME.err will be keep on growing
>> > on the bason host login.toolforge.org
>> >
>> > How to avoid this?
>> >
>> > Thanks.
>> >
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Security and maintenance release: 1.31.15 / 1.35.3 / 1.36.1

[Sam Reed]

I would like to announce the release of MediaWiki 1.31.15, 1.35.3 and
1.36.1!

These releases also serve as a maintenance release for these branches.

While tarballs have already been uploaded, git tags will follow later on
today.

An "MediaWiki Extensions Security Release Supplement" email will follow
this one.

It is also noted that the patch from MediaWiki 1.35.2 to MediaWiki 1.35.3
(mediawiki-1.35.3.patch.gz/mediawiki-1.35.3.patch.zip) is larger than
expected due to an issue with the initial importing of skins and extensions
as submodules into the REL1_35 branch; similar to what happened for 1.36.0.
A bug has been filed for this issue, but doesn't cause any practical issues
otherwise.

== Security fixes ==
* (T280226, CVE-2021-35197) SECURITY: Prevent blocked users from purging
pages.

== Links to all mentioned tasks ==
* https://phabricator.wikimedia.org/T280226

== Release notes ==

Full release notes for 1.31.15:
https://phabricator.wikimedia.org/diffusion/MW/browse/REL1_31/RELEASE-NOTES-1.31
https://www.mediawiki.org/wiki/Release_notes/1.31

Full release notes for 1.35.3:
https://phabricator.wikimedia.org/diffusion/MW/browse/REL1_35/RELEASE-NOTES-1.35
https://www.mediawiki.org/wiki/Release_notes/1.35

Full release notes for 1.36.1:
https://phabricator.wikimedia.org/diffusion/MW/browse/REL1_36/RELEASE-NOTES-1.36
https://www.mediawiki.org/wiki/Release_notes/1.36

For information about how to upgrade, see


**
Download:
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.tar.gz
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.zip

Download without bundled extensions:
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-core-1.31.15.tar.gz
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-core-1.31.15.zip

Patch to previous version (1.31.14):
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.patch.gz
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.patch.zip

GPG signatures:
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-core-1.31.15.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-core-1.31.15.zip.sig
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.zip.sig
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.patch.gz.sig
https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.15.patch.zip.sig

Public keys:
https://www.mediawiki.org/keys/keys.html

**
Download:
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.tar.gz
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.zip

Download without bundled extensions:
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.3.tar.gz
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.3.zip

Patch to previous version (1.35.2):
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.patch.gz
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.patch.zip

GPG signatures:
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.3.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-core-1.35.3.zip.sig
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.zip.sig
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.patch.gz.sig
https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.3.patch.zip.sig

Public keys:
https://www.mediawiki.org/keys/keys.html

**
Download:
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.tar.gz
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.zip

Download without bundled extensions:
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-core-1.36.1.tar.gz
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-core-1.36.1.zip

Patch to previous version (1.36.0):
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.patch.gz
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.patch.zip

GPG signatures:
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-core-1.36.1.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-core-1.36.1.zip.sig
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.tar.gz.sig
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.zip.sig
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.patch.gz.sig
https://releases.wikimedia.org/mediawiki/1.36/mediawiki-1.36.1.patch.zip.sig

Public keys:
https://www.mediawiki.org/keys/keys.html
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to 

[Wikitech-l] Thank you Tuesday

[Vira Motorko]

Many thanks to everyone involved! 珞

ср, 23 черв. 2021, 01:43 користувач Amir Sarabadani 
пише:

> Hi,
> Lots of people thanked me for deploying mailman3 but I want to mention
> that it would have not been possible without Wikimedia Cloud Services team
> giving a lot of resources to me so I could have a test setup and puppetize
> mailman3 easily which in turn made deployment to production much easier.
>
> Thank you for providing such a critical infrastructure to us! Keep up the
> great work.
>
> Best
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Feature idea: data structure to improve translation capabilities

[petr . kadlec]

Hi!

On Wed, Jun 23, 2021 at 11:05 AM Wolter HV  wrote:

> [2021-06-20 17:43 +0100] John:
> > Off hand isn’t this something that wikidata was setup to handle?
>
> I'm not sure, but I don't see the functionality currently being there in
> Wiktionary.  Is this something currently under development?
>

Yeah, for something like fifteen years, I guess… :-) See e.g. OmegaWiki
(formerly known as WiktionaryZ).

The modern incarnation of machine-readable dictionary is the
Lexicographical Data project on Wikidata. It is a nice project, definitely
go take a look at it, but it is not really an evolution/improvement of
Wiktionary but rather a fresh start. (Among other reasons because of the
license incompatibility of Wiktionary’s CC-BY-SA with Wikidata’s CC-0.) See
https://www.wikidata.org/wiki/Wikidata:Lexicographical_data

-- [[cs:User:Mormegil | Petr Kadlec]]
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Why does the train start on Tuesday?

[Dan Garry (Deskana)]

On Tue, 22 Jun 2021 at 23:10, Jon Robson  wrote:

> Apologies if I didn't make myself clear, but it seems I didn't given both
> Amir's comments. I am very happy that we have these, and my question was *not
> *why do we have them, but rather* why do we only have 2*. I want more of
> them and every time I've asked why we don't have more, I've been told it's
> a community decision and that seems odd to me.
>

It's a community decision in the sense that it wasn't something the WMF
decided for them, it was those communities decided it for themselves.
They're open to it probably as a result of being generally less
conservative when it comes to changes, and more willing to try things out
early. But, it's not a community decision in the sense that you, as staff,
can't get involved and try to recruit more wikis!

So, if other wikis are asked if they want to be in group 1, understand the
advantages and disadvantages of that, and they said yes, I doubt anyone
would raise any objections. Having more wikis in that group to test things
on early would definitely be a benefit.

Amir's suggestion to contact wikis that have been open to Growth features,
like Czech and Korean (I think?), sounds like a great one, especially if
there are community ambassadors that could help explain what it means to be
in group 1.

Dan
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Feature idea: data structure to improve translation capabilities

[Wolter HV]

[2021-06-20 17:43 +0100] John:
> Off hand isn’t this something that wikidata was setup to handle?

I'm not sure, but I don't see the functionality currently being there in
Wiktionary.  Is this something currently under development?

Regards,
Wolter HV
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Need help to login with pwb.py with a bot account

[JJMC89]

Hello T.Shrinivasan,

The relevant portion is {'result': 'Failed', 'reason': 'Incorrect username
or password entered. Please try again.'}.

You should not be using the same password that is used to log in to the web
UI. For a BotPassword, see the configuration steps at
https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords.

Regards,
JJ

On Sat, Jun 19, 2021 at 9:00 AM Shrinivasan T 
wrote:

> Hello all,
>
> I am trying to run a pywikibot based bot on toolforge server.
>
> i logged into tool forge.
>
> Cloned the pywikibot.
>
> created the user-config.py and user-password.py with below content.
>
> cat user-config.py
> mylang = 'pa'
> family = 'wikisource'
> usernames['wikisource']['pa'] = 'WD-WS Integration Bot'
> usernames['wikidata']['wikidata'] = 'WD-WS Integration Bot'
> password_file = "user-password.py"
>
>
>
> cat user-password.py
> ('WD-WS Integration Bot', BotPassword('wd-ws-integration-bot', 'PASSWORD
> HERE'))
>
> Created bot account at pa.wikisource.org
>
> https://pa.wikisource.org/wiki/User:WD-WS_Integration_Bot
>
> Can login to the pa.wikisource.org with the same user name and
> credentials.
>
> But, cant login using the command line using pwb.py.
>
> Getting the below error.
>
> tools.wdwsbot@tools-sgebastion-07:~/pywikibot-core$ python3
> /data/project/shared/pywikibot/stable/pwb.py login
> WARNING: No user is logged in on site wikisource:pa
> Logging in to wikisource:pa as WD-WS Integration Bot@wd-ws-integration-bot
> Traceback (most recent call last):
>  File
> "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core_stable/pywikibot/data/api.py",
> line 2931, in login_to_site
>response['messagecode'] == 'login-throttled'
> KeyError: 'messagecode'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>  File "/data/project/shared/pywikibot/stable/pwb.py", line 365, in 
>if not main():
>  File "/data/project/shared/pywikibot/stable/pwb.py", line 360, in main
>file_package)
>  File "/data/project/shared/pywikibot/stable/pwb.py", line 74, in
> run_python_file
>main_mod.__dict__)
>  File "/data/project/shared/pywikibot/stable/scripts/login.py", line
> 151, in 
>main()
>  File "/data/project/shared/pywikibot/stable/scripts/login.py", line
> 139, in main
>site.login(autocreate=autocreate)
>  File
> "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core_stable/pywikibot/site/_apisite.py",
> line 388, in login
>if login_manager.login(retry=True, autocreate=autocreate):
>  File
> "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core_stable/pywikibot/login.py",
> line 307, in login
>self.login_to_site()
>  File
> "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core_stable/pywikibot/data/api.py",
> line 2939, in login_to_site
>'handled:\n{}'.format(response))
> pywikibot.exceptions.CaptchaError: Captcha encountered which cannot be
> handled:
> {'result': 'Failed', 'reason': 'Incorrect username or password
> entered. Please try again.'}
> CRITICAL: Exiting due to uncaught exception  'pywikibot.exceptions.CaptchaError'>
>
>
> Need help to solve this.
>
> Thanks.
>
>
> --
> Regards,
> T.Shrinivasan
>
>
> My Life with GNU/Linux : http://goinggnu.wordpress.com
> Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com
>
> Get Free Tamil Ebooks for Android, iOS, Kindle, Computer :
> http://FreeTamilEbooks.com
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: How to get alerts on errors for cron jobs on toolforge

[QEDK]

I do remember getting emails (last year?) from running cronjobs on the grid
until I disabled them, iirc it was when they failed, but not necessarily a
description of the error. I literally cannot find any documentation
regarding this anymore so I almost have to think if I didn't imagine this
entire ordeal.

Best,
QEDK

On Wed, Jun 23, 2021, 09:01 Sam Wilson  wrote:

> I think the only way is to have your cronjob do the emailing of the
> output, and then delete (or move) the output file so it's not repeated
> on the next run. Something like:
>
> #!/bin/bash
>
> ## Call actual job script here.
> ./do-tool-things.sh
>
> ## Email output to maintainers.
> echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain;
> charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out |
> /usr/sbin/exim -odf -i tools.tooln...@tools.wmflabs.org
>
> ## Remove output file.
> rm /data/project/toolname/cron-tools.jobname.out
>
> Although of course the details might differ for different tools.
>
>
> On 22/6/21 4:52 pm, Shrinivasan T wrote:
> > Hello all,
> >
> > I have a bot at toolforge, running weekly once in cron job using "jsub"
> >
> > Followed the below link for the setup.
> > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
> >
> > Getting the JOBNAME.out and JOBNAME.err files at the home folder.
> >
> > Is there any other way like email notification to get alerts, if some
> > error happens?
> >
> > I think the files JOBNAME.out and JOBNAME.err will be keep on growing
> > on the bason host login.toolforge.org
> >
> > How to avoid this?
> >
> > Thanks.
> >
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Why does the train start on Tuesday?

[Jaime Crespo]

On Wed, Jun 23, 2021 at 12:10 AM Jon Robson  wrote:

> I understand the Friday is a buffer, but it's not a great buffer,
> particularly now
>

I don't have any suggestions, as I am not super-familiar with the current
situation, but I can understand (and suffered) from some hidden bugs (not
immediately obvious) taking some time to surface and/or reach the right
people, but I have some questions:

* How often are issues surfaced in the group0 -> group1 vs group1 ->
group2, are there any stats to back the need for a change there?
* Without changing the actual deploying days or the frequency, would there
be any benefit of shifting the deploy over multiple weeks? (random example
Tu: group1->group2, (new branch) We: group0, Th: group0-> group1) or would
that make things worse?
* You mention commons. I am guessing Commons, and Wikidata, to some extent-
are both large sites with a lot of visibility but also very different from
the core features that are similar to most other wikis, but the test
version of those on group0 may not be enough to catch all issues. Is there
something that could be improved specifically for those sites?
* Can we do something to improve the speed from "a user notices an issue
with the site" to "the right team/owner is aware of it and acts on it"?

-- 
Jaime Crespo

___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/