[Wikitech-l] Testing via GitHub actions

[Jeroen De Dauw]

Hey,

Has anyone created a GitHub action to run tests for a MediaWiki extension
yet?

I'd like to run the PHPUnit tests for some of my extensions via GitHub
actions. Unfortunately this is not as simple as in the typical PHP project,
since doing composer install + vendor/bin/phpunit does not cut it. It'd be
fantastic if someone already created a clean action to do this that I can
copy.

Best

--
Jeroen De Dauw | www.EntropyWins.wtf 
Professional wiki hosting and services: www.Professional.Wiki

Entrepreneur | Software Architect | Open Source | Longtermism
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] npm and security (was: Re: TechCom Radar 2020-07-01)

[Aron Manning]

Hello,

On Tue, 7 Jul 2020 at 00:30, Kunal Mehta  wrote:

>
> > hosting a *private node package repository* in some form,
> > typically in a git repository, where only *vetted versions* of packages
> are
> > checked in.
>
> In theory this addresses the problems, but I think the biggest problem
> is just the volume and quality of code that needs reviewing.
>

I've suspected vetting and quantity poses big challenges. To my surprise I
haven't found any information on how that's done.
To discuss this topic, I've made the subtask to the build step RFC:
T257072  "Determine Node package
auditing workflows"


> Here's what the actual diff looks like:
> .
>

Analyzing all that information is a superhuman task. A gerrit/gitlab like
review interface would make it more approachable.
The way Pnpm stores packages (uncompressed in a git repo) would enable
that. Yarn stores .zip files, but a separate uncompressed repo could be
used for code review, updates submitted as regularly reviewed patches.


> I don't believe it's possible to review that much code on a regular
> basis, reacting to the speed at which many npm packages move. We could
> stop upgrading all the time, but that would effectively be forking and
> IMO put us in a worse position.
>

100% review wouldn't be sustainable IMO (causing developer burnout very
quickly), but looking for specific patterns exhibited in malicious packages
could be a successful approach to increase trust in the audited code.
Patterns like:
* An unmaintained repo receiving an update, which is a common solution to
inject malicious code.
* New packages added to the dependency tree.

An interesting article in this regard:
https://portswigger.net/daily-swig/new-npm-scanning-tool-sniffs-out-malicious-code
I wonder if the npm-scan  tool
mentioned therein has been evaluated.
The repo of former malicious packages (npm-zoo
) is also worth mentioning.
I've collected a few notable incidents in the RFC under section
A_few_examples_of_NPM_incidents

.


> I also note that it's impossible to review just the git changelog of a
> package, because the npm maintainer can upload any arbitrary tarball of
> code to npm, whether or not it matches the git repo. (This isn't
> exclusive to npm, pypi, crates.io suffer from this problem too.
> composer/packagist doesn't though.)
>

A tool looking for differences between the git repo and the npm tarball
could be useful.
It's possible though that many packages would require special treatment if
the tarball isn't simple to map on the git repo.
However, just a simple check to see if a new npm release has a
corresponding git tag or release - or any commits at all - would catch
injections done with a stolen NPM token.

How do these alternative package managers address the quantity of npm
> packages installed that need review?
>

None of the package managers can or intend to do code review apart from
`npm audit`, available with all 3.
It seems to me there is an expectation that PMs will protect us. It should
be clarified that no tool can do that, the purpose of these tools is to
give 100% control over what packages and versions are installed.
What these PMs provide is detailed in the RfC for evaluating alternative PMs


What versions we add to the local package repo is up to us. Separate
ticket: T257072 

I think a 2 stage deployment process would subject packages to as much
scrutiny as possible within the constraints:

1. An auditing package repository with all the updates to be vetted. This
would be used in sandboxed environments to expose updates to developers,
who could notice outstanding behavior, warning signs.
2. A stable package repository for CI and not sandboxed environments.

The review process:

1. The auditing repo only includes packages used in WMF projects.
2. Package versions need to be greenlighted for auditing too. This is
preceded by a basic check of the validity of that version to look for eg,
stolen credential injections, but code is only reviewed if suspicious, eg.
new packages, unexpected updates.
3. Package versions would stay in this stage for some time (eg. 2 weeks),
depending on the package and urgency.
4. A changelog in the auditing repo tracks the newest updates, informing
developers about what packages to pay attention to.
5. One of the developers dedicated to package vetting does a deeper review
of the code. This should be aided by heuristic tools.
6. When confidence in a version is built, that version is greenlighted for
the 

Re: [Wikitech-l] Gerrit SSH host keys will change on July 14th

[Daniel Zahn]

> that's around 11:00 UTC.

Correction: My bad, that is actually 16:00 UTC.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Gerrit SSH host keys will change on July 14th

[Daniel Zahn]

This is a heads-up that we are planning to replace the host keys
for the Gerrit SSH server at gerrit.wikimedia.org:29418.

The change is planned for Tuesday, July 14th in the PDT
morning right after the MediaWiki train, that's around 11:00 UTC.

(https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_July_14)

The RSA key will be replaced with a longer version and additionally we
will start to offer ecdsa_256, ecdsa_384, ecdsa_521 and ed25519.

The service will not be RSA-only anymore which some users had already
reported as an issue.

After the change on Gerrit, your git / git-review / direct ssh
commands are expected to fail with errors about mismatched or changed
host keys or host identification.

This is expected.
You will need to remove the old, no longer used host key, and verify
the new one.

To remove the old host key, follow the instructions on screen or
consult the manual of your SSH software. Once that is done, retry the
command, and you'll be prompted to verify the new host key.

You can find the new keys for verification in this email below and on
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418

If they match, confirm, and your command should continue. Once you
have successfully updated the host key you should no longer see any
errors.

If you are running any bots talking to gerrit-ssh please also update
their configuration accordingly and restart where needed.

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418

ssh_host_rsa_key
2048 SHA256:j9/pXXc9WzjQwYP0t7nlzqH9EBOTw6q7DgcfnamJtsY
gerrit-code-rev...@gerrit1001.wikimedia.org (RSA)

ssh_host_ecdsa_256_key
256 SHA256:58swSiByT+4LVqs30/FqJpEPj+Mwjtn3WJY5hitlEgM
gerrit-code-rev...@gerrit1001.wikimedia.org (ECDSA)

ssh_host_ecdsa_384_key
384 SHA256:vFEVzNGuagPmYiw9EIwBStzd0X+gtprZzOi8vbLxAfc
gerrit-code-rev...@gerrit1001.wikimedia.org (ECDSA)

ssh_host_ecdsa_521_key
521 SHA256:OWb1uenhapK7AFPfEB+NRxgfxhktZ1Q6C5eCy+VbgsY
gerrit-code-rev...@gerrit1001.wikimedia.org (ECDSA)

ssh_host_ed25519_key
256 SHA256:njCmWMsshq3MqQxyIFO36UNwCwzTamXERqylF1XJhd8
gerrit-code-rev...@gerrit1001.wikimedia.org (ED25519)


-- 
Daniel Zahn 
Operations Engineer

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Hard deprecation of the Revision class

[Gergo Tisza]

The Revision -> RevisionRecord switch was a key part of both making
MediaWiki dependency-injection-friendly and making it aware of multiple
content slots per revision. It's great to see us following through on a
large-scale refactoring like that. Huge thanks to DannyS712 (who as done
awesome work on a number of other deprecations too), the Core Platform
Team, and everyone else who has been working on it!
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] CI and Code Review

[Kaartic Sivaraam]

Hi Gregg,

Thanks for the details.


>
>PS: Please let me know where else this announcement should be sent.
>

I think it would be a good idea to send this to the Wikitech Ambassadors 
 list.

If not already done, mentioning in Tech News 
[https://meta.m.wikimedia.org/wiki/Tech/News] would be a good thing too.

-- 
Sivaraam

Sent from my Android device with K-9 Mail. Please excuse my brevity.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] CI and Code Review

[Adam Wight]

I feel positive about every aspect of this announcement.  I've enjoyed 
my own experiments with GitLab and its integrated CI. It's a huge relief 
that we'll be able to move towards a feature-branch strategy.


Even as a paid developer with 8 years of wiki experience, Gerrit is 
still an obstacle to my work and a challenge to use "correctly".  
Similarly, tinkering with our custom CI framework has been fun and 
rewarding, but I understand it's less fun for the small circle of people 
doing the daily maintenance.


The only detail which makes me nervous is that GitLab seems to lack 
diversity on their Board and upper management, so they may be prone to 
diseases endemic to Silicon Valley such as sudden corporate takeover or 
a shift to more aggressive profiteering. Hopefully self-hosting will 
insulate ourselves from this kind of turmoil, or at least gives us a few 
years of runway to migrate away as needed.


Kind regards,
Adam W.

On 7/6/20 7:39 PM, Greg Grossmeier wrote:

First, apologies for not announcing this last week. A short work week
coupled with a new fiscal year delayed this until today.

tl;dr: Wikimedia will be moving to a self-hosted (in our datacenter(s))
GitLab Community Edition (CE) installation for both code review and
continuous integration (CI).

Longer:
We are making a change to our code review and continuous integration (CI)
systems in response to a complex set of inputs (Developer Satisfaction
Survey[0], passing comments/critiques, an evaluation of replacement
continuous integration infrastructure[1], feedback from leaders in the
Foundation, etc) and evaluation conversations with Wikimedia Technology
department leadership (eg CTO, VPs) and representatives from Wikimedia SRE,
Architecture, Core Platform, Product, Security, and Technical Engagement
teams. In those conversations with Technology department leadership,
coordinated by our CTO Grant Ingersoll, we determined that an RFC was not
needed for this decision[2].

We plan to replace Gerrit and Zuul+Jenkins (the software that powers our CI
system) with GitLab. We hope that making a move to GitLab now will address
most of the concerns and desires that are able to be addressed via software
alone.

We join a growing list of other free/open knowledge organizations using a
self-hosted GitLab installation and look forward to the added benefits of
working together with them.

The project portal page lives at:
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/GitLab
It is a living documentation portal and we will continue to update it as we
go. The Talk: page is also available for your feedback and questions (and
is already being used). We hope everyone will join in to make this
transition as successful as possible.

Notably, I would like to point people to the conversation[3] about
evaluating pull-request style workflows and finding a recommended workflow
for our projects. While pull-request workflows are much more common in the
wider software development world{{cn}} we would like to provide as much
guidance as reasonable so that we are using our tools to the best of their
ability.

Here is the list of stakeholders as we have them now. In a RACI[4] model
these would be Consulted. These are the people the Wikimedia Release
Engineering team will be consulting with most closely as they get this
going.
* SRE/Service Ops - Mark and delegate(s)
* Security - Chase P and Scott B
* Core Platform Team - TBD
* Technical Engagement - TBD
* Product - Daniel C

“TBD” means we have asked for a representative and we’re waiting to hear
back on confirmation. We also have a few other non-WMF groups that we have
already reached out to or will be shortly to include in that list; feel
free to ping me with other suggestions.

What does this mean for you as you do your work today? Right now, nothing.
But as we set up the new GitLab installation we will be looking for early
adopters to give us feedback as we work on improvements. As we start to
migrate more users and repositories we will strive to help everyone as much
as possible and reasonable. It should go without saying that this includes
completely volunteer projects using the shared infrastructure.

The full timeline and plan will be posted to the above project portal page.

For the avoidance of doubt, this does not impact issue/task management;
that will remain in Phabricator.

Thank you,

Greg

PS: Please let me know where else this announcement should be sent.

[0] https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2020
[1]
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG
[2] see also:
https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Charter#Areas_within_scope
[3] https://www.mediawiki.org/wiki/Topic:Vpbwawb4lkdy89ym
[4] https://en.wikipedia.org/wiki/Responsibility_assignment_matrix
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org

Re: [Wikitech-l] Hard deprecation of the Revision class

[Daniel Kinzler]

It's great to see this finally happening!

A big THANK YOU to DannyS712, Petr, and everyone involved! The Revision class
had been "soft" deprecated every since we introduced RevisionStore and
RevisionRecord in 1.31. Getting the necessary work scheduled for removing
hundreds of usages proved difficult. Now DannyS712 just dove in and tackled the
beast. Thank you again! It's great to see volunteers take on tasks like this.
Open Source rocks, and our community is awesome!

-- daniel

Am 06.07.20 um 22:18 schrieb Petr Pchelko:
> Hey all,
> 
> TLDR: your extension CI might break due to hard deprecation of Revision
> class.
> 
> Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
> Revision class.
> Loads of work has been done to prepare for this moment, mostly by a
> volunteer developer DannyS712 with code review
> support from almost every corner of the MediaWiki developer universe. You
> can judge the amount of work by the number
> of subtasks in the tracking ticket[2]
> 
> A lot of effort has been done to update all the WMF deployed extensions and
> some non-deployed ones to the new code,
> In accordance with stable interface policy deprecation section[3]. However
> due to the gravity of a change, we might have
> missed some corner cases. Missed usages will not cause problems in
> production since the only consequence of using hard
> deprecated code is a log message, but some CI tests might start failing.
> If you find your tests failing, please replace all the usages of the
> Revision class according to 1.35 release notes,
> and your tests should start passing again. In case you’re having troubles
> doing it, please reach out to core platform team
> and we will try to help.
> 
> This is a huge milestone in decoupling MediaWiki core! We understand that
> this might cause some inconveniences, apologize
> for them and are hoping we can help with resolving any issues.
> 
> Cheers.
> Petr Pchelko
> 
> 1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
> 2. https://phabricator.wikimedia.org/T246284
> 3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> 

-- 
Daniel Kinzler
Principal Software Engineer, Core Platform
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Hard deprecation of the Revision class

[Kosta Harlan]

Thank you to everyone who worked on this, and especially to DannyS712, your
work is valued & appreciated!

Kosta
--
Kosta Harlan (he/him)
Sr Software Engineer
Wikimedia Foundation


On Mon, Jul 6, 2020 at 10:19 PM Petr Pchelko  wrote:

> Hey all,
>
> TLDR: your extension CI might break due to hard deprecation of Revision
> class.
>
> Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
> Revision class.
> Loads of work has been done to prepare for this moment, mostly by a
> volunteer developer DannyS712 with code review
> support from almost every corner of the MediaWiki developer universe. You
> can judge the amount of work by the number
> of subtasks in the tracking ticket[2]
>
> A lot of effort has been done to update all the WMF deployed extensions and
> some non-deployed ones to the new code,
> In accordance with stable interface policy deprecation section[3]. However
> due to the gravity of a change, we might have
> missed some corner cases. Missed usages will not cause problems in
> production since the only consequence of using hard
> deprecated code is a log message, but some CI tests might start failing.
> If you find your tests failing, please replace all the usages of the
> Revision class according to 1.35 release notes,
> and your tests should start passing again. In case you’re having troubles
> doing it, please reach out to core platform team
> and we will try to help.
>
> This is a huge milestone in decoupling MediaWiki core! We understand that
> this might cause some inconveniences, apologize
> for them and are hoping we can help with resolving any issues.
>
> Cheers.
> Petr Pchelko
>
> 1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
> 2. https://phabricator.wikimedia.org/T246284
> 3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l