[Wikitech-l] Re: Proposal: new structure for MediaWiki RELEASE-NOTES

[Riccardo Coccioli]

On Tue, Jul 20, 2021 at 3:38 AM Petr Pchelko  wrote:

> *Alternative solutions*
>
> We could write a custom merge driver for RELEASE-NOTES which always puts
> ‘ours’ before ’theirs’,
> but I’m not sure the result will justify the investment.
>

Probably overkill for MediaWiki but I'd like to mention the way that Python
developers (CPython) manages their release notes, in case it might be
useful.
The TL;DR is that each change has a unique valid reStructuredText file in a
specific directory and then there is a tool to merge all changes when a
release is made.

The full process from a contributor point of view is described in [1].
The tool used to both generate the change files and merge them into a
release file is [2].

[1]
https://devguide.python.org/committing/#updating-news-and-what-s-new-in-python
[2] https://pypi.org/project/blurb/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Re: [Wikitech-l] Migrating away from nose (was 2017-11-01 Scrum of Scrums meeting notes)

[Riccardo Coccioli]

I've migrated Cumin from 'nose' to 'pytest' a while ago, here's a
summary of my experience.

First of all we need to distinguish the two main parts of Python
tests: writing the tests and collecting and running them.

In the classical setup, 'unittest' would be used for writing the tests
and 'nose' for collecting and running them, often wrapped in a call to
'coverage'.
To use 'pytest' just as a test collector and runner, basically means
replacing 'nose' with 'pytest' and 'coverage' with the 'pytest-cov'
module and adding 'pytest-runner' to the 'setup_requires' in setup.py.
There are some caveats though, depending how many nose-specific
features are used and if any unsupported idiom is used, see also [1]
and [2].

Given the status of 'nose', I've looked at 'nose2', but it didn't seem
mature enough and actively developed. When I looked at it few months
ago that was no commit in the last 6-months and the latest release is
from mid-2016. I've seen that there has been some recent activity
though.
I then chose 'pytest' as I got the impression that the Python
community is slowly moving towards it. In the process I've also got
some quick and friendly help on their issue tracker.

I've done this migration in Cumin primarily because I wanted to
overcome some limitations in the previous setup, namely:
- more compact output of passing tests, listing dots per file and not
naming them all
- more verbose output on failing tests with the stacktrace, the code
involved and an easy way to collect separately stderr and stdout
- coverage plugin works with multiprocess testing (very useful for
integration tests) while nose+coverage did not work (known limitation)

Once this step is done, the other, optional but somehow natural step,
is to use 'pytest' also for writing the tests, replacing 'unittest'.
This could be partly automated by a tool like [3], although in my
experience it doesn't cover all the cases and needs some manual
cleanup, but surely helps.
This allows to take full advantage of all the pytest features, as some
are not compatible with unittest-based tests.

I've done this migration for the integration tests of Cumin to take
advantage of 'pytest' parametrization feature [4] that was very useful
in my case for integration tests that were automatically generated.

Once that was done, I've decided to migrate everything to 'pytest' because:
- coherence, to not have integration tests written with 'pytest' and
unit test written with 'unittest'
- to be able in the future to use advanced features of pytest in the
unittests, because they will most likely not work in unittest.TestCase
subclasses (and indeed it happened)
- style-wise 'unittest' doesn't follow PEP8

The main style-related drawback of using 'pytest' for unit tests is
that all the 'self.assertSomeThing(foo)' must be replaced by standard
asserts like 'assert foo == something', and the assert lines don't go
nicely multiline because is not possible to use parentheses. But this
can be easily avoided just assigning the expected and test result
values to variables just before doing the assert.

If you have specific questions while migrating your project, feel free
to contact me [5] or add me as code reviewer on Gerrit (user: Volans).

Riccardo

[1] https://docs.pytest.org/en/latest/nose.html
[2] https://docs.pytest.org/en/latest/unittest.html
[3] https://github.com/pytest-dev/unittest2pytest
[4] https://docs.pytest.org/en/latest/parametrize.html
[5] https://meta.wikimedia.org/wiki/User:RCoccioli_(WMF)


On Wed, Nov 1, 2017 at 7:34 PM, Greg Grossmeier  wrote:
>
> (adding in qa@ list)
>
> 
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA512
> >
> > Hi,
> >
> > On 11/01/2017 10:53 AM, Grace Gellerman wrote:
> > > === Scoring Platform === ** Converted our tests from nosetests to
> > > pytest (Anyone else making the transition?) *** nosetests is
> > > under-maintained *** python community is moving towards pytest ***
> > > converting our repos was pretty straightforward
> >
> > I didn't realize nose was in such a state :( They put a notice[1] on
> > their website, but I always skipped it.
> >
> > pytest[2] claims that it's compatible with nose out of the box - was
> > that the experience you had as well?
> >
> > Off the top of my head, Pywikibot, CI, and other miscellaneous
> > projects all use nose and should probably be migrated over to pytest.
> >
> > [1] https://nose.readthedocs.io/en/latest/#note-to-users
> > [2] https://docs.pytest.org/en/latest/
> >
> > - -- Legoktm
> > -BEGIN PGP SIGNATURE-
> >
> > iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAln6D0gXHGxlZ29rdG1A
> > bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KKbMw/+Nh/tGRagrvZgJqHBZakvvAYE
> > IHCUFPTUhfQ4cQ++oUqDpblAxR/FNO6BN2UQIrIh9gUylHCf/hwZ3BToC5GGPFY5
> > D9Sgi/ce9v5dp3IIvNZq/NZgb0DkityVkxWg8tiZJPwhJ6hTKw9LjaPl++ee9Po/
> > 940WYy7UQ7L/xwgy7rNJrViW0j4VD4mM7+PVvxPuv6rovOpa/H0VMbSlEuoXQYFG
> > P2DPJE1f8B9tn/xun6O6tveWdD7cdna2DVCXhFF+Tldnpa+XtOerE3P+YBA8Gp/+
> > 

Re: [Wikitech-l] [Wikipedia-l] Data centre switchover to Eqiad

[Riccardo Coccioli]

Those variables are in the process to be moved to Etcd to make the
configuration dynamic, see [1] and related tasks, hence (re)moving the
problem in the first place. Even in the case that this will not go to
production in time, the hardcoded part in Switchdc can be easily removed,
please find my detailed reply directly in the task in [2].

Riccardo

[1] https://phabricator.wikimedia.org/T156924
[2] https://phabricator.wikimedia.org/T164177#3224771

On Mon, May 1, 2017 at 1:37 AM, Eddie Greiner-Petter <
wikimedia@eddie-sh.de> wrote:

> Hey,
> I had a quick chat with Zppix about how to solve this and we ended up
> creating this task instead of just sending in a patch:
>
> https://phabricator.wikimedia.org/T164177
>
> While it looks quite trivial to change the strings in the config files
> on the first glance, I noticed that the switchdc script actually
> hardcodes this message in the scripts used to set mediawiki ro and back
> rw (maybe also somewhere else, haven't checked), which needs us to pay
> attention to not break switchdc by exchanging that message. Details are
> described on the task, let's continue there.
>
> - Eddie
>
> On 01.05.2017 00:18, Jaime Crespo wrote:
> > On Sun, Apr 30, 2017 at 9:24 PM, Eddie Greiner-Petter <
> > wikimedia@eddie-sh.de> wrote:
> >
> >> That reminds me that we noticed during switch to codfw that the message
> >> shown when trying to really edit a page (the mediawiki read-only
> >> message) contains:
> >>
> >> The system administrator who locked it offered this explanation:
> >> MediaWiki is in read-only mode for maintenance. Please try again in a
> >> few minutes
> >>
> >> which isn't quite informative. Is there a task for changing the "offered
> >> explanation" part? Some hint about the DC switch (and maybe a link to
> >> the meta page) would be better.
> >
> >
> > The read-only messages are controlled by these strings:
> >
> > https://phabricator.wikimedia.org/source/mediawiki-config/
> browse/master/wmf-config/db-codfw.php;a65f35adbc9d2c8c9a85e956a64661
> 783d2c973d$645
> > https://phabricator.wikimedia.org/source/mediawiki-config/
> browse/master/wmf-config/db-eqiad.php;a65f35adbc9d2c8c9a85e956a64661
> 783d2c973d$664
> >
> > I think with the pressure of keeping everything up, plus discussing
> > internally if we should declare a specific amount of time (given it
> doesn't
> > auto-update) we ended up a very generic message. We are, however, looking
> > at showing better error messages like on the ticket I reported at
> > https://phabricator.wikimedia.org/T163455#3199813
> >
> > Please send a pull request or file a new ticket on Phabricator with the
> > #operations and #codfw-rollout tags with a proposal and we can definitely
> > change it by Wednesday.
> >
> > Thanks,
> >
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The codfw deployment server is Naos

[Riccardo Coccioli]

= TL;DR
- The codfw deployment server is now Naos: naos.codfw.wmnet [1]. Its
SSH fingerprints are available at [2].
- Mira is no longer a deployment server.
- Naos is also the current active deployment server until May 3rd [3], [4].
- As always, please use the DNS record deployment.eqiad.wmnet to know
which deployment server is the active one.


= Detailed version:
Few days before the datacenter switchover to codfw performed last week
[3], [4], with perfect timing, the existing codfw deployment server
Mira manifested hardware issues.
Being Mira an old server, also out of warranty, a new deployment
server in the codfw datacenter was deployed and configured.

As a result, from now on, the deployment server for the codfw
datacenter is Naos: naos.codfw.wmnet [1]. You can find its SSH
fingerprints in [2].

As part of the datacenter switchover procedures [4], also the
deployment server was switched to codfw, hence Naos is currently the
active deployment server, until the switchback to eqiad as master
datacenter, scheduled on May 3rd [3], [4].
The DNS entries for both deployment.eqiad.wmnet and
deployment.codfw.wmnet always point to the active deployment server
and should be used to verify which one is the active one.

As always, a large message of the day (MOTD) is shown to alert people
to not use the passive deployment server on login, and it is now shown
on both Tin and Mira.

I've also updated all the references to Mira that I've found on
Wikitech, but please verify if you have any direct reference to it
that might need to be updated.

Thanks,
Riccardo

[1] https://wikitech.wikimedia.org/wiki/Naos
[2] https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/naos.codfw.wmnet
[3] https://blog.wikimedia.org/2017/04/18/codfw-temporary-editing-pause/
[4] https://wikitech.wikimedia.org/wiki/Switch_Datacenter

-- 
Riccardo Coccioli
Software Engineer, Technical Operations
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l