Re: [Wikitech-l] Dealing with composer dependencies in early MediaWiki initialization

2019-06-25 Thread Bryan Davis 
On Tue, Jun 25, 2019 at 8:21 PM Kunal Mehta  wrote:
>
> I see 3 main ways to move forward:
>
> 1. Move vendor/autoload.php earlier in Setup.php, potentially breaking
> extensions that still rely on composer autoloading for initialization.
> 2. Set $wgServer = false or something in DefaultSettings.php, and then
> fill it in later in Setup.php *after* the composer autoloader has been
> loaded, potentially breaking anyone relying on the value of $wgServer
> in LocalSettings.php.
> 3. (status quo) not librarize code that runs before composer
> autoloader initialization. :(

There may be more entanglements here than I'm seeing, but I think that
there may be an option 4: add code in WebRequest to replace the use of
IP::splitHostAndPort() and IP::combineHostAndPort().

IP::combineHostAndPort() is trivial, and I think that
splitHostAndPort() could be replaced with a semi-clever call to
parse_url() that looked something like:

  $parts = parse_url( 'fake://' . $_SERVER[$varName] );

Bryan
-- 
Bryan Davis  Wikimedia Foundation
[[m:User:BDavis_(WMF)]] Manager, Technical EngagementBoise, ID USA
irc: bd808v:415.839.6885 x6855

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Dealing with composer dependencies in early MediaWiki initialization

2019-06-25 Thread Kunal Mehta 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

I (with Reedy's help) recently started work on librarizing MediaWiki's
IP class into a separate composer package (wikimedia/ip-utils[1]). The
main motivation was so that the Parsoid PHP port could use it[2].

However, I ran into an unexpected hitch[3], as it seems we're using
the IP class before the composer autoloader is even intialized. Here's
the basic initialization in Setup.php:

- - AutoLoader.php (MediaWiki's)
- - Defines.php
- - DefaultSettings.php
  - $wgServer = WebRequest::detectServer()
- Calls IP::splitHostAndPort()
- - GlobalFunctions.php
- - vendor/autoload.php (composer's)

My understanding is that composer's autoloader runs late so extensions
registering themselves using it can add their stuff to the necessary
globals.

And we call WebRequest::detectServer() in DefaultSettings.php so that
in LocalSettings.php people can use the value of $wgServer for other
stuff.

I see 3 main ways to move forward:

1. Move vendor/autoload.php earlier in Setup.php, potentially breaking
extensions that still rely on composer autoloading for initialization.
2. Set $wgServer = false or something in DefaultSettings.php, and then
fill it in later in Setup.php *after* the composer autoloader has been
loaded, potentially breaking anyone relying on the value of $wgServer
in LocalSettings.php.
3. (status quo) not librarize code that runs before composer
autoloader initialization. :(

Advice/input welcome.

[1] https://packagist.org/packages/wikimedia/ip-utils
[2]
https://gerrit.wikimedia.org/g/mediawiki/services/parsoid/+/77064cfff717
6493a2828bb4f95f397dfce7d659/src/Utils/Title.php#46
[3] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/519089/

- -- Legoktm
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEE2MtZ8F27ngU4xIGd8QX4EBsFJpsFAl0S1oQACgkQ8QX4EBsF
Jpufrg/+J9RUUxRAtgJLEkyACE6GREis0eyEIZnWmMr3s9YpFPoqtWocFrUk6Wsn
W7d9Oda/8CW0/d894gGMn8LWIj9oWq2gMPWzCVFpg8uu3r4967qxBp+ba29uMOJw
Qpw6DhXtPvVAeUCy8P38Y5vM7TGmV+J1T5jDY21zimT1dRrJsI1KD+u/Ue3nYy/y
B1ic3i7vJfhYErdhHgN98ETXfXOaDx4rgd2N7PLjVNx3IYCC8LNiR8wSLuydfdbk
PLTT1bA2qi0h2wgcEr7Qtq9YstVotq8899rgKLtGDBwQi3qGNcdOgQGEMFDVfjfO
CsiWocj6s4oc3ScVj+Eb9xtvIqhNx+oRbWE1vKd4TmtSdyzpv6xadV60tq5qNFEY
I0cBDOWU5UFNHbvbyjK4dqIDEVhJ6LiEgLVBOj81U27s8mR4Dv/yFB3eac0ROk7p
gaEeOjfhtVU558XfpEsmu1H05VJT3kXNxK8y0UQOjy11SErzsXv6vDzyzLDJM/W7
WF0I4nyjeqVsBjLBN9li+5AnU3cAKVOCfZ+/aRYyg89Du//nJRjm+4lxnuPrGlaG
ES/nVUnkDZ9Yc/xA1yacm3Ytx9hpoY1mIZgxxxveyeU1KsNXAZ2BOGA2T7kU4yUw
Uyg+byYwI+1uVOjAVd3BInGV2R2/GmeIn9FOpthBaw8wcz0Y/8c=
=tU4+
-END PGP SIGNATURE-

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit HTTP Token Auth re-enabled

2019-06-25 Thread Tyler Cipriani 

I spoke too soon :(

Frustratingly, Gerrit 2.15.14 exacerbated an existing bug[0] to the 
extent that I feel like we have no choice but to rollback to 2.15.13.


I have re-disabled HTTP token auth for the time being.

Apologies for the false hope,
-- Tyler

[0]. 


On 19-06-24 15:53:25, Tyler Cipriani wrote:

Hi all!

tl;dr: Gerrit HTTP token auth has been re-enabled. To use it you'll need to
generate a token via your preferences page[0].

Gerrit HTTP token auth was disabled in mid-march due to concerns about its
implementation[1].

Thanks to the work of Paladox and Gerrit upstream in Gerrit 2.15.14[2] we've
re-enabled HTTP token authentication.

I previously removed all HTTP auth tokens, so in order to use HTTP token auth
you'll need to generate a fresh token via your preference page[0]

Your Lowly Gerrit Fiddler,
-- Tyler

[0]. 
[1]. 
[2]. 





signature.asc
Description: PGP signature
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Finding and fixing cyclic dependencies

2019-06-25 Thread Antoine Musso 

Guten Tag,

Time flow, and I forgot to reply to that email :-\

On 13/05/2019 23:11, Daniel Kinzler wrote:

Hey Antoine!

Am 13.05.19 um 20:59 schrieb Antoine Musso:

Hello,

A few months ago, the code health group enquired about the Php metrics static
analysis tool [1] and it has been rather straightforward to run it automatically
and publish its report:

https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/

The report has a list of potential coupling and also show relation between
objects.  That might be a complement.


Yea, that's what I used for my initial analysis for the session on decoupling at
TechConf in Portland :)

phpmetrics and similar tools are useful, but they do not analyze the transitive
dependencies (at least not sufficiently, for my use case). That is, they can't
tell me whether two classes are coupled, but they can't tell me which classes to
decouple to resolve clusters of tightly coupled code. And they can't measure how
"tangled" the codebase is overall, just how "good" or "bad" a given class is (or
all classes are, on average).


Oh so yeah that is going way deeper in the graph analysis. I would guess 
someone with a background in graph theory would be able to help on that 
front.  I love the topic, but not enough to actually learn it :-\




As for your tool, I am pretty sure we can easily run it automatically and
publish it next to the PHP Metrics report?   The addition to CI has been rather
straightforward:

A container:
https://gerrit.wikimedia.org/r/#/c/integration/config/+/469689/

The Jenkins job:
https://gerrit.wikimedia.org/r/#/c/integration/config/+/469690/


Ruprecht comes with a few dependencies that may make containerization less
straight forward. Not terribly hard, but somewhat annoying. E.g. it needs python
1 *and* 3, it needs graph-tool which has to be installed as a debian package
from a non-standard repo, etc.

If you feel like looking into that, I'd of course be happy, of course ;)


There is no python 1?  I am assuming you are referring to python2.7. 
They can be coinstalled in the same Debian based container, they simply 
are different binaries: /usr/bin/python2 /usr/bin/python3. Should be easy.


Depends on what a non standard repo is? The CI images are build with the 
non-standard repository apt.wikimedia.org, we have some PHP packages 
from sury.org (the person who also package PHP for Debian) and others. 
It is super easy, has long as the packages are signed with a gpg key.



If you could please fill a task against #continuous-integration-config 
in Phabricator with a rough breakdown of how to setup the stack. I am 
sure it can be done rather quickly.


--
Antoine "hashar" Musso


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Discovery Weekly Update for the week starting 2019-06-10

2019-06-25 Thread Chris Koerner 
Hello,
This is the weekly update from the Search Platform team for the week
starting 2019-06-10.

As always, feedback and questions are welcome.

== Discussions ==

=== Search ===
* Trey has finished his first analysis of a proposal to strip
diacritics in Slovak for searching. Slovak searchers don't always use
them, but removing them causes new problems for stemming. Read more on
MediaWiki or Phabricator —and join the discussion if you speak Slovak!
[0] [1]
* Mathew worked on creating a Cookbook to restart WDQS after applying
updates or doing a deployment [2]
* The team worked with Analytics to port usage of
mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request [3]
* When a new item is created, we have functionality to instant-index
it synchronously, even if with partial information, so it appears
quickly in the index - Stas worked on getting the un-redirected items
to be instant-indexed [4]
* CirrusSearch\SearcherTest::testSearchText PHPUnit tests take a long
time to run and is slowing down change processing in CI - David helped
to make it faster [5]
* It is currently not possible to evaluate all items without labels,
so we investigated adding haslabel:all and determined that matching an
_all field and a descriptions_all field worked well [6]
* We removed old settings ($wgLexemeUseCirrus and
$wgLexemeDisableCirrus) as they were both set to true for production
(no longer applicable / out of date) [7]
* David fixed an issue where suggested article are no longer shown,
API Usage for the morelike search endpoint has dropped since train
deploy of 1.34.0-wmf.7 -- by accident. [8]
* Geodata not being returned for some files - David merged in a few
patches to be sure that we don't lose CoordinatesOutput when multiple
slots are available [9]
* An old request to fix where Special:Search doesn't use labels and
descriptions for suggestions but just the item ID was worked on by
Stas and was enabled [10]
* A PHP notice in MW-Vagrant when searching with CirrusSearch was
happening in the action API and web serach interface. Thanks for
fixing it, Ottomata!

[0] 
https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Folding_Diacritics_in_Slovak
[1] https://phabricator.wikimedia.org/T223787#5260102
[2] https://phabricator.wikimedia.org/T221832
[3] https://phabricator.wikimedia.org/T68
[4] https://phabricator.wikimedia.org/T206256
[5] https://phabricator.wikimedia.org/T225184
[6] https://phabricator.wikimedia.org/T224611
[7] https://phabricator.wikimedia.org/T225183
[8] https://phabricator.wikimedia.org/T224879
[9] https://phabricator.wikimedia.org/T223767
[10] https://phabricator.wikimedia.org/T55652



Subscribe to receive on-wiki (or opt-in email) notifications of the
Discovery weekly update.

https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly

The archive of all past updates can be found on MediaWiki.org:

https://www.mediawiki.org/wiki/Discovery/Status_updates

Interested in getting involved? See tasks marked as "Easy" or
"Volunteer needed" in Phabricator.

[1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R
[2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R

Yours,
Chris Koerner (he/him)
Community Relations Specialist
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Tech Talks] June 25, 2019, 6 PM UTC, Just what is Analytics doing back there?

2019-06-25 Thread Nuria Ruiz 
The talk has started, this is the you tube stream:
https://www.youtube.com/watch?reload=9=GD0PEDFysfM


Thanks,

Nuria

On Mon, Jun 10, 2019 at 8:53 AM Subramanya Sastry 
wrote:

> Hi Everyone,
>
> It's time for Wikimedia Tech Talks 2019 Episode 5!
> This month's talk will take place *June 25, 2019 at 6:00 PM UTC.*
>
> *Topic*: Just what is Analytics doing back there?
>
> *Speaker*: Dan Andreescu, Senior Software Engineer, Analytics
>
> *Summary*: We take care of twelve systems. Data flows through them to
> answer the many questions that our community and staff have about
> our piece of the open knowledge movement. Let's take a look at how
> these systems fit together and dive deeper into some of the more
> interesting algorithms.
>
> *YouTube stream for viewers*: https://www.youtube.com/watch?v=GD0PEDFysfM
>
> During the live talk, you are invited to join the discussion on IRC
> at #wikimedia-office
>
> You can watch past Tech Talks here:
> https://www.mediawiki.org/wiki/Tech_talks
>
> If you are interested in giving your own tech talk, you can learn more
> here:
>
> https://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event#Tech_talks
>
>
> Subbu.
> (Standing in for Sarah Rodlund who is currently away)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Tech Talks] June 25, 2019, 6 PM UTC, Just what is Analytics doing back there?

2019-06-25 Thread Subramanya Sastry 

Reminder: Talk starts in 5 mins.  -Subbu.

On 6/24/19 9:31 AM, Subramanya Sastry wrote:


Reminder: This talk is tomorrow. -Subbu.

On 6/10/19 10:52 AM, Subramanya Sastry wrote:

Hi Everyone,

It's time for Wikimedia Tech Talks 2019 Episode 5!
This month's talk will take place *June 25, 2019 at 6:00 PM UTC.*

*Topic*: Just what is Analytics doing back there?

*Speaker*: Dan Andreescu, Senior Software Engineer, Analytics

*Summary*: We take care of twelve systems. Data flows through them to
answer the many questions that our community and staff have about
our piece of the open knowledge movement. Let's take a look at how
these systems fit together and dive deeper into some of the more
interesting algorithms.

*YouTube stream for viewers*: 
https://www.youtube.com/watch?v=GD0PEDFysfM


During the live talk, you are invited to join the discussion on IRC
at #wikimedia-office

You can watch past Tech Talks here: 
https://www.mediawiki.org/wiki/Tech_talks


If you are interested in giving your own tech talk, you can learn 
more here:
https://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event#Tech_talks 



Subbu.
(Standing in for Sarah Rodlund who is currently away)

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Future CI planning, feedback requested

2019-06-25 Thread Lars Wirzenius 
In the Release Engineering team we're preparing for a new CI system.
The current one needs to be replaced. It works well, but parts of it
are getting obsolete. In particular, the Zuul version we use is
obsoleted by upstream. The new version of Zuul is entirely different.
Because of this, we are taking the opportunity to re-think the whole
approach to CI. We would like to introduce the possibility of
continuous delivery and deployment, in addition to continuous
intergration.

Earlier this year, we started a working group to evaluate candidates
for software. In phase 1, we set up some criteria for evaluation, and
considered a large number of possibilities, and winnowed the list down
to three candidates: GitLab CI, Zuul v3, and Argo. For details and a
report, see [0].

We are currently writing up what the new CI system should look like in
more detail. The approach taken is to start with what's needed and
wanted, rather than what the tools provide. The document has had a
first round of internal review, to get rid of the worst issues, and v2
is now open for feedback from the whole movement. You can find it at
[1]. Those with a WMF Google account can comment directly on the doc,
everyone else please use email, either by responding to this email via
wikitech-l or directly to me.

[0] 
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG
[1] 
https://docs.google.com/document/d/1EQuInEV-eY_5kxOZ8E1qEdLr8fb6ihwOD9V_tpVFWuU/edit

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Thank You Tuesday

2019-06-25 Thread Lucas Werkmeister 
I’m pretty sure I’ve said this before, even in another Thank you Tuesday
thread, but once again thank you to Legoktm and everyone else working
on MediaWiki
code search ! It continues to be an
invaluable tool when planning refactorings on code that might be used
elsewhere.

Thank you also to the Foundation’s Trust and Safety team, and the Code of
Conduct Committee.

Cheers,
Lucas

Am Di., 25. Juni 2019 um 16:13 Uhr schrieb Zppix :

> I wanted to restart the thank you Tuesday thread for this week. So heres
> my list of thanks:
>
> Paladox, and releng, for all you do for maintaining Gerrit, Phabricator,
> and CI so we can keep making so many needed changes to mediawiki and other
> software/projects.
>
> Andre, for sorting through hundreds if not thousands of tickets every day,
> without him, I’m sure phab would be a mess of unsorted tasks!
>
> Finally, thanks to all the volunteers and staff that are constantly
> deploying, and creating patches to make MediaWiki and related projects as
> bug-free and user friendly as possible.
>
>
>
> --
> Devin “Zppix” CCENT
> Volunteer Wikimedia Developer
> Africa Wikimedia Developers Member and Mentor
> Volunteer Mozilla Support Team Member (SUMO)
> Quora.com Partner Program Member
> enwp.org/User:Zppix
> **Note: I do not work for Wikimedia Foundation, or any of its chapters. I
> also do not work for Mozilla, or any of its projects. **
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Lucas Werkmeister (he/er)
Full Stack Developer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
https://wikimedia.de

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Thank You Tuesday

2019-06-25 Thread Zppix 
I wanted to restart the thank you Tuesday thread for this week. So heres my 
list of thanks:

Paladox, and releng, for all you do for maintaining Gerrit, Phabricator, and CI 
so we can keep making so many needed changes to mediawiki and other 
software/projects.

Andre, for sorting through hundreds if not thousands of tickets every day, 
without him, I’m sure phab would be a mess of unsorted tasks!

Finally, thanks to all the volunteers and staff that are constantly deploying, 
and creating patches to make MediaWiki and related projects as bug-free and 
user friendly as possible.



--
Devin “Zppix” CCENT
Volunteer Wikimedia Developer
Africa Wikimedia Developers Member and Mentor
Volunteer Mozilla Support Team Member (SUMO)
Quora.com Partner Program Member
enwp.org/User:Zppix
**Note: I do not work for Wikimedia Foundation, or any of its chapters. I also 
do not work for Mozilla, or any of its projects. ** 
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit HTTP Token Auth re-enabled

2019-06-25 Thread Zppix 
Echoing Brian, Thanks for the hard work once again paladox, and releng, your 
work does not go unnoticed. Keep it up! 

--
Devin “Zppix” CCENT
Volunteer Wikimedia Developer
Africa Wikimedia Developers Member and Mentor
Volunteer Mozilla Support Team Member (SUMO)
Quora.com Partner Program Member
enwp.org/User:Zppix
**Note: I do not work for Wikimedia Foundation, or any of its chapters. I also 
do not work for Mozilla, or any of its projects. ** 

> On Jun 24, 2019, at 7:59 PM, Bryan Davis  wrote:
> 
>> On Mon, Jun 24, 2019 at 3:53 PM Tyler Cipriani  
>> wrote:
>> 
>> Hi all!
>> 
>> tl;dr: Gerrit HTTP token auth has been re-enabled. To use it you'll need to
>> generate a token via your preferences page[0].
>> 
>> Gerrit HTTP token auth was disabled in mid-march due to concerns about its
>> implementation[1].
>> 
>> Thanks to the work of Paladox and Gerrit upstream in Gerrit 2.15.14[2] we've
>> re-enabled HTTP token authentication.
>> 
>> I previously removed all HTTP auth tokens, so in order to use HTTP token auth
>> you'll need to generate a fresh token via your preference page[0]
>> 
>> Your Lowly Gerrit Fiddler,
>> -- Tyler
>> 
>> [0]. 
>> [1]. 
>> [2]. 
> 
> Thank you for the update Tyler and thank you to everyone who worked to
> clear the security concerns with the feature.
> 
> I do not use it often, but being able to push patches to Gerrit from
> an untrusted location (like a project local Puppet master in a Cloud
> VPS project) with this workflow is pretty nice:
> * Generate a fresh password at
> https://gerrit.wikimedia.org/r/#/settings/http-password
> * Git push to gerrit over https with username/password auth
> * Regenerate a password at
> https://gerrit.wikimedia.org/r/#/settings/http-password to invalidate
> the password that was exposed to the untrusted instance/network
> 
> Bryan
> -- 
> Bryan Davis  Wikimedia Foundation
> [[m:User:BDavis_(WMF)]] Manager, Technical EngagementBoise, ID USA
> irc: bd808v:415.839.6885 x6855
> 
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l