Re: [Wikitech-l] Hard deprecation of the Revision class

[Brian Wolff]

Hard deprecated = generates warnings if $wgEnableDevelopmentWarnings is on
(which in turn cause unit tests to fail, as warnings are considered errors
in a test). The code is still there and working for the moment.

--
Brian

On Monday, July 6, 2020, Jay R. Ashworth  wrote:

> 
> I think you might be better calling that "removed"; the entire *point*
> of the jargon usage of 'deprecated' is "it's still there and working,
> but start moving off it soon, cause we're gonna remove it".
> 
>
> - Original Message -
> > From: "Petr Pchelko" 
> > To: "Wikimedia developers" 
> > Sent: Monday, July 6, 2020 4:18:34 PM
> > Subject: [Wikitech-l] Hard deprecation of the Revision class
>
> > Hey all,
> >
> > TLDR: your extension CI might break due to hard deprecation of Revision
> > class.
> >
> > Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
> > Revision class.
> > Loads of work has been done to prepare for this moment, mostly by a
> > volunteer developer DannyS712 with code review
> > support from almost every corner of the MediaWiki developer universe. You
> > can judge the amount of work by the number
> > of subtasks in the tracking ticket[2]
> >
> > A lot of effort has been done to update all the WMF deployed extensions
> and
> > some non-deployed ones to the new code,
> > In accordance with stable interface policy deprecation section[3].
> However
> > due to the gravity of a change, we might have
> > missed some corner cases. Missed usages will not cause problems in
> > production since the only consequence of using hard
> > deprecated code is a log message, but some CI tests might start failing.
> > If you find your tests failing, please replace all the usages of the
> > Revision class according to 1.35 release notes,
> > and your tests should start passing again. In case you’re having troubles
> > doing it, please reach out to core platform team
> > and we will try to help.
> >
> > This is a huge milestone in decoupling MediaWiki core! We understand that
> > this might cause some inconveniences, apologize
> > for them and are hoping we can help with resolving any issues.
> >
> > Cheers.
> > Petr Pchelko
> >
> > 1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
> > 2. https://phabricator.wikimedia.org/T246284
> > 3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land
> Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] npm and security (was: Re: TechCom Radar 2020-07-01)

[Kunal Mehta]

Hi,

On 2020-07-02 02:41, Aron Manning wrote:
> Recently, I haven't seen the security question of npm being discussed on
> https://phabricator.wikimedia.org/T199004
> and thought that question was not on topic. If that's not the case: is
> there another discussion I'm not aware of?

There's been some work around protecting developers from npm packages,
especially  but I don't believe
there are any active public discussions.

> The issue of package security has been answered by the Node community
> multiple times in different forms through the years.
> Were these solutions evaluated? These generally avoid unvetted code by
> hosting a *private node package repository* in some form,
> typically in a git repository, where only *vetted versions* of
packages are
> checked in.

In theory this addresses the problems, but I think the biggest problem
is just the volume and quality of code that needs reviewing.

Take 
for example. It bumps 3 packages: stylelint-config-wikimedia (owned by
us), webpack (minor version bump), and markdown-to-jsx (patch bump). The
latter two are security issues (whether they're actually exploitable in
our context is another discussion).

Here's what the actual diff looks like:
.

There's at least a thousand lines changed, not including the
minified/compiled code that's impossible to review. And that doesn't
even show the npm packages that download compiled binaries on installation.

(Feel free to throw other patches at the libup-diff tool, but it's alpha
quality still, I hadn't fixed it up enough yet to announce it.)

I don't believe it's possible to review that much code on a regular
basis, reacting to the speed at which many npm packages move. We could
stop upgrading all the time, but that would effectively be forking and
IMO put us in a worse position.

I also note that it's impossible to review just the git changelog of a
package, because the npm maintainer can upload any arbitrary tarball of
code to npm, whether or not it matches the git repo. (This isn't
exclusive to npm, pypi, crates.io suffer from this problem too.
composer/packagist doesn't though.)

> These include the complete dependency trees, the package hashes and the
> full source code of packages, therefore provide more complete security
than
> libraryupgrader2, which - in my understanding - only controls our
top-level
> dependencies.

npm dependency trees should be fully locked via package-lock.json. libup
currently only upgrades top-level dependencies, but it also runs `npm
audit fix` in response to npm security advisories (see the current
listing: ).

But honestly I consider libup's "npm audit fix" mode just damage control
at this point. If we can't trust the code we're running, then we're more
likely to protect ourselves by fixing the security issues we do know about.

In short, if you've been running npm install/test/etc. on your machine,
I would consider any ssh/GPG/etc. keys that it could've accessed
compromised. Krinkle goes into this much better in his blog post:
.

> Without completeness I'd mention 2 recent solutions:
> 

How do these alternative package managers address the quantity of npm
packages installed that need review?

> 2. I've been using these PMs and made the necessary additions to some of
> our projects. Where to discuss my findings and is there someone interested
> in reviewing patches adding the missing dependencies to 'package.json'
> files, that would make the repositories compatible with these PMs without
> altering npm's behavior?

First I'd start with documenting the current problems we face with npm
and its ecosystem and once we have agreement on that, then beginning to
look for solutions.

Ultimately, I think it's important to remember that npm is a proprietary
service run by a for-profit company (formerly npm inc., now Github).
We're always going to be fighting against them with little ability to
cause significant change. I think a free software, community based
registry, like practically every other major language, could do so much
better in this area.

-- Legoktm



signature.asc
Description: OpenPGP digital signature
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Hard deprecation of the Revision class

[Jay R. Ashworth]

I think you might be better calling that "removed"; the entire *point*
of the jargon usage of 'deprecated' is "it's still there and working,
but start moving off it soon, cause we're gonna remove it".


- Original Message -
> From: "Petr Pchelko" 
> To: "Wikimedia developers" 
> Sent: Monday, July 6, 2020 4:18:34 PM
> Subject: [Wikitech-l] Hard deprecation of the Revision class

> Hey all,
> 
> TLDR: your extension CI might break due to hard deprecation of Revision
> class.
> 
> Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
> Revision class.
> Loads of work has been done to prepare for this moment, mostly by a
> volunteer developer DannyS712 with code review
> support from almost every corner of the MediaWiki developer universe. You
> can judge the amount of work by the number
> of subtasks in the tracking ticket[2]
> 
> A lot of effort has been done to update all the WMF deployed extensions and
> some non-deployed ones to the new code,
> In accordance with stable interface policy deprecation section[3]. However
> due to the gravity of a change, we might have
> missed some corner cases. Missed usages will not cause problems in
> production since the only consequence of using hard
> deprecated code is a log message, but some CI tests might start failing.
> If you find your tests failing, please replace all the usages of the
> Revision class according to 1.35 release notes,
> and your tests should start passing again. In case you’re having troubles
> doing it, please reach out to core platform team
> and we will try to help.
> 
> This is a huge milestone in decoupling MediaWiki core! We understand that
> this might cause some inconveniences, apologize
> for them and are hoping we can help with resolving any issues.
> 
> Cheers.
> Petr Pchelko
> 
> 1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
> 2. https://phabricator.wikimedia.org/T246284
> 3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Hard deprecation of the Revision class

[Petr Pchelko]

Hey all,

TLDR: your extension CI might break due to hard deprecation of Revision
class.

Today a Gerrit change[1] has been merged, hard deprecating MediaWiki core
Revision class.
Loads of work has been done to prepare for this moment, mostly by a
volunteer developer DannyS712 with code review
support from almost every corner of the MediaWiki developer universe. You
can judge the amount of work by the number
of subtasks in the tracking ticket[2]

A lot of effort has been done to update all the WMF deployed extensions and
some non-deployed ones to the new code,
In accordance with stable interface policy deprecation section[3]. However
due to the gravity of a change, we might have
missed some corner cases. Missed usages will not cause problems in
production since the only consequence of using hard
deprecated code is a log message, but some CI tests might start failing.
If you find your tests failing, please replace all the usages of the
Revision class according to 1.35 release notes,
and your tests should start passing again. In case you’re having troubles
doing it, please reach out to core platform team
and we will try to help.

This is a huge milestone in decoupling MediaWiki core! We understand that
this might cause some inconveniences, apologize
for them and are hoping we can help with resolving any issues.

Cheers.
Petr Pchelko

1. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/608845
2. https://phabricator.wikimedia.org/T246284
3. https://www.mediawiki.org/wiki/Stable_interface_policy#Deprecation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 1.35 Branch Cut and Release

[Evan Prodromou]

Hey, all. This is the roughly 1-week "pencils down" warning for the
upcoming MediaWiki 1.35 release branch cut.

The branch cut is scheduled to be made on 14 July 2020 at 02:00 UTC.

(That's late on 13 July in the Americas and parts of the Pacific. Timezones
are like magic, aren't they?)

After that point, new features won't be merged into 1.35; the master branch
will become part of the 1.36 release.

If you have tickets that are tagged for `mw-1.35-release`, please finish
them, untag them, or reach out to get them resolved.*

After the branch cut, bug fixes will be manually ported, but feature
changes won't.

Thanks for your work on and interest in MediaWiki.

-Evan

* See https://phabricator.wikimedia.org/tag/mw-1.35-release/ for details.




On Tue, Jun 2, 2020 at 8:58 AM Cindy Cicalese 
wrote:

> On March 30, we announced that we were temporarily pushing back the release
> of MediaWiki 1.35 due to uncertainty resulting from the COVID-19 pandemic.
> We are now ready to begin the process of moving forward with the release.
>
> The first step is cutting the release branch, REL1_35, which will occur on
> Monday, July 13.
>
> We will then be in "pencils down" mode: developers will stop targeting
> MediaWiki 1.35 for new features. Instead, any new features would continue
> to be applied to master and would target MediaWiki 1.36 or later. The only
> work that would continue towards MediaWIki 1.35 would be blockers -
> critical bug fixes or features close to completion that need to make it
> into the release. This would happen by merging those patches into master
> and then backporting them to the REL1_35 branch.
>
> We anticipate that MediaWiki 1.35 will be released at the beginning of
> August.
>
> We appreciate your patience in these difficult times. Wishing you safety
> and health,
>
> Cindy
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] CI and Code Review

[Greg Grossmeier]

First, apologies for not announcing this last week. A short work week
coupled with a new fiscal year delayed this until today.

tl;dr: Wikimedia will be moving to a self-hosted (in our datacenter(s))
GitLab Community Edition (CE) installation for both code review and
continuous integration (CI).

Longer:
We are making a change to our code review and continuous integration (CI)
systems in response to a complex set of inputs (Developer Satisfaction
Survey[0], passing comments/critiques, an evaluation of replacement
continuous integration infrastructure[1], feedback from leaders in the
Foundation, etc) and evaluation conversations with Wikimedia Technology
department leadership (eg CTO, VPs) and representatives from Wikimedia SRE,
Architecture, Core Platform, Product, Security, and Technical Engagement
teams. In those conversations with Technology department leadership,
coordinated by our CTO Grant Ingersoll, we determined that an RFC was not
needed for this decision[2].

We plan to replace Gerrit and Zuul+Jenkins (the software that powers our CI
system) with GitLab. We hope that making a move to GitLab now will address
most of the concerns and desires that are able to be addressed via software
alone.

We join a growing list of other free/open knowledge organizations using a
self-hosted GitLab installation and look forward to the added benefits of
working together with them.

The project portal page lives at:
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/GitLab
It is a living documentation portal and we will continue to update it as we
go. The Talk: page is also available for your feedback and questions (and
is already being used). We hope everyone will join in to make this
transition as successful as possible.

Notably, I would like to point people to the conversation[3] about
evaluating pull-request style workflows and finding a recommended workflow
for our projects. While pull-request workflows are much more common in the
wider software development world{{cn}} we would like to provide as much
guidance as reasonable so that we are using our tools to the best of their
ability.

Here is the list of stakeholders as we have them now. In a RACI[4] model
these would be Consulted. These are the people the Wikimedia Release
Engineering team will be consulting with most closely as they get this
going.
* SRE/Service Ops - Mark and delegate(s)
* Security - Chase P and Scott B
* Core Platform Team - TBD
* Technical Engagement - TBD
* Product - Daniel C

“TBD” means we have asked for a representative and we’re waiting to hear
back on confirmation. We also have a few other non-WMF groups that we have
already reached out to or will be shortly to include in that list; feel
free to ping me with other suggestions.

What does this mean for you as you do your work today? Right now, nothing.
But as we set up the new GitLab installation we will be looking for early
adopters to give us feedback as we work on improvements. As we start to
migrate more users and repositories we will strive to help everyone as much
as possible and reasonable. It should go without saying that this includes
completely volunteer projects using the shared infrastructure.

The full timeline and plan will be posted to the above project portal page.

For the avoidance of doubt, this does not impact issue/task management;
that will remain in Phabricator.

Thank you,

Greg

PS: Please let me know where else this announcement should be sent.

[0] https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2020
[1]
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG
[2] see also:
https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Charter#Areas_within_scope
[3] https://www.mediawiki.org/wiki/Topic:Vpbwawb4lkdy89ym
[4] https://en.wikipedia.org/wiki/Responsibility_assignment_matrix
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Moving from Gerrit to GitLab

[Physikerwelt]

Hi all,

thank you for this well documented and clearly thought through
decision. Will there be some kind of code review que integrated to the
system or will this stay in phabricator. I mean, the best code review
system does not help if contributed code is not reviewed. I understand
that this is a serious amount of work, but probably investing
resources in doing the code review would be a good complement to the
implementation of a new code review system. For instance

 https://phabricator.wikimedia.org/project/view/4614/

is lists 41/10 incoming tasks. For me, personally that means that I
have not continued developing for several weekends since I am still
waiting for code review. In the long run this is somehow frustrating
since simple adoptions to the new API standards (developed by WMF)
take months and in the end  there is no room for new new features at
all.

All the best

pyhsikerwelt


On Mon, Jul 6, 2020 at 11:56 AM Antoine Musso  wrote:
>
> Le 06/07/2020 à 10:36, Federico Leva (Nemo) a écrit :
> >
> > (I also note that the page references something called "OKR" which was
> > not previously introduced to this list, as far as I know, including a
> > specific line which is not found in any public document. Can we link a
> > definition of what OKR means in the context of WMF, and ideally also
> > some public document containing the referenced line? I suppose it would
> > be some kind of annual or quarterly plan or something like that.)
>
> Hello Federico,
>
> OKR stands for "Objectives and Key Results", it is a management
> framework to keep track of objectives and their achievements (=
> "outcomes").  So that each department, team, individuals plan ambitious
> objectives and resulting outcomes which are assessed at the end of a period.
>
> If I remember properly the concept originates from Intel back in the
> 1980's and has later being adopted by Google which has let them grow
> dramatically. It eventually has spread to the technology scene in the US
> (LinkedIn, Uber etc).
>
> The first occurrence for us is probably the 2015 article which stated
> OKR was mean to be applied for fiscal year 2015 and onward:
> https://www.mediawiki.org/wiki/Wikimedia_Engineering/OKR
>
> Folks will correct me if needed, but I think the framework got formally
> introduced to the whole foundation for fiscal year 2019-2020.
> Previously we had annual and quarterly goals which is essentially the
> same thing: write down what is planned and commit to do it.
>
> You can read a short introduction by Google at:
> https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/
>
> :)
>
> --
> Antoine "hashar" Musso
>
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Moving from Gerrit to GitLab

[Antoine Musso]

Le 06/07/2020 à 10:36, Federico Leva (Nemo) a écrit :
> 
> (I also note that the page references something called "OKR" which was
> not previously introduced to this list, as far as I know, including a
> specific line which is not found in any public document. Can we link a
> definition of what OKR means in the context of WMF, and ideally also
> some public document containing the referenced line? I suppose it would
> be some kind of annual or quarterly plan or something like that.)

Hello Federico,

OKR stands for "Objectives and Key Results", it is a management
framework to keep track of objectives and their achievements (=
"outcomes").  So that each department, team, individuals plan ambitious
objectives and resulting outcomes which are assessed at the end of a period.

If I remember properly the concept originates from Intel back in the
1980's and has later being adopted by Google which has let them grow
dramatically. It eventually has spread to the technology scene in the US
(LinkedIn, Uber etc).

The first occurrence for us is probably the 2015 article which stated
OKR was mean to be applied for fiscal year 2015 and onward:
https://www.mediawiki.org/wiki/Wikimedia_Engineering/OKR

Folks will correct me if needed, but I think the framework got formally
introduced to the whole foundation for fiscal year 2019-2020.
Previously we had annual and quarterly goals which is essentially the
same thing: write down what is planned and commit to do it.

You can read a short introduction by Google at:
https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/

:)

-- 
Antoine "hashar" Musso




___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Moving from Gerrit to GitLab

[Federico Leva (Nemo)]

To understand the parameters of this discussion, it would be useful to
know whether the Wikimedia Technical Committee Charter

is still in force.

Grant, maybe you can answer?

(I also note that the page references something called "OKR" which was
not previously introduced to this list, as far as I know, including a
specific line which is not found in any public document. Can we link a
definition of what OKR means in the context of WMF, and ideally also
some public document containing the referenced line? I suppose it would
be some kind of annual or quarterly plan or something like that.)

Federico

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l