Re: [Wikitech-l] Logs are down

[יגאל חיטרון]

​Hi, Moriel. Good idea. I will.
Igal.
​


2017-09-13 19:56 GMT+03:00 Moriel Schottlender 
:

> I think this can be solved easily if you just add the URL of what you're
> looking at when you send an email or report such things.
>
> This will help everyone understand the context even if you aren't aware of
> all the other products or tools that we have.
> You need to understand that sending an email saying that any sort of logs
> or sql tables are down makes people very very concerned, especially during
> a weekend. It is common courtesy to add the context, to make sure people
> don't jump in panic over the wrong problem. And it will be beneficial to
> actually solving the problem you report - because people will be able to
> look into the correct problem, rather than wondering if you mean something
> else that you might not be aware exist.
>
> Just add the link you're looking at, that should solve at least some of the
> confusion for the future :)
>
> (It's also a good idea in general to anyone reporting an issue, be it here
> or in Phabricator, and will save developers a lot of time trying to find
> the issue themselves)
>
> Moriel
>
> On Tue, Sep 12, 2017 at 7:20 AM, יגאל חיטרון 
> wrote:
>
> > The problem is I have no idea what is Toolforge and what is Beta Cluster.
> > And I never thought there is more than one option. If I would know, of
> > cause, I whould give all the details.
> > Igal
> >
> >
> > 2017-09-12 17:17 GMT+03:00 Brad Jorsch (Anomie) :
> >
> > > On Tue, Sep 12, 2017 at 5:51 AM, יגאל חיטרון 
> wrote:
> > >
> > > > Hi. I think you missed that. I started from "The sql log tables are
> > > dead".
> > > > It shows exactly where is the problem.
> > > >
> > >
> > > The criticism that you are receiving is that you didn't specify *which*
> > log
> > > tables in your email. It turns out you meant the replicas in Toolforge,
> > but
> > > your original message was so vague that it could easily have been
> > > indicating a problem in the Beta Cluster or on the production wikis.
> > >
> > > In general, it's best to say explicitly what you're talking about
> instead
> > > of assuming people will somehow know it.
> > >
> > >
> > > --
> > > Brad Jorsch (Anomie)
> > > Senior Software Engineer
> > > Wikimedia Foundation
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Logs are down

[Moriel Schottlender]

I think this can be solved easily if you just add the URL of what you're
looking at when you send an email or report such things.

This will help everyone understand the context even if you aren't aware of
all the other products or tools that we have.
You need to understand that sending an email saying that any sort of logs
or sql tables are down makes people very very concerned, especially during
a weekend. It is common courtesy to add the context, to make sure people
don't jump in panic over the wrong problem. And it will be beneficial to
actually solving the problem you report - because people will be able to
look into the correct problem, rather than wondering if you mean something
else that you might not be aware exist.

Just add the link you're looking at, that should solve at least some of the
confusion for the future :)

(It's also a good idea in general to anyone reporting an issue, be it here
or in Phabricator, and will save developers a lot of time trying to find
the issue themselves)

Moriel

On Tue, Sep 12, 2017 at 7:20 AM, יגאל חיטרון  wrote:

> The problem is I have no idea what is Toolforge and what is Beta Cluster.
> And I never thought there is more than one option. If I would know, of
> cause, I whould give all the details.
> Igal
>
>
> 2017-09-12 17:17 GMT+03:00 Brad Jorsch (Anomie) :
>
> > On Tue, Sep 12, 2017 at 5:51 AM, יגאל חיטרון  wrote:
> >
> > > Hi. I think you missed that. I started from "The sql log tables are
> > dead".
> > > It shows exactly where is the problem.
> > >
> >
> > The criticism that you are receiving is that you didn't specify *which*
> log
> > tables in your email. It turns out you meant the replicas in Toolforge,
> but
> > your original message was so vague that it could easily have been
> > indicating a problem in the Beta Cluster or on the production wikis.
> >
> > In general, it's best to say explicitly what you're talking about instead
> > of assuming people will somehow know it.
> >
> >
> > --
> > Brad Jorsch (Anomie)
> > Senior Software Engineer
> > Wikimedia Foundation
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] TechCom IRC discussion tonight: Job Queue Issues

[Daniel Kinzler]

Hi all!

This is a quick reminder that tonight, at the TechCOm IRC hour, we will be
talking about the job queue. There have been several issues iwth it lately, and
we want to make sure that we have all relevant aspects on the radar.

As always, the discussion will take place in the IRC channel
#wikimedia-office on Wednesday 21:00 UTC (2pm PDT, 23:00 CEST).


This is not an RFC meeting, as there is no concrete proposal. Rather, it's an
opportunity to further our understanding of the problems and hand, and to float
ideas for possible improvements.

I have prepared a quick brain dump of my current understanding of the job queue
issues .
Here's a copy for your convenience, but please comment directly in the document.


Observations:

* Latest instance of the JQ exploding: https://phabricator.wikimedia.org/T173710

* With 600k jobs in the backlog of commonswiki, only 7k got processed in a day.

* For wikis with just a few thousand pages, we sometimes see millions of
UpdateHtmlCache jobs sitting in the queue.

* Jobs that were triggered months ago were found to continue failing and 
re-trying


Issues and considerations:

* Jobs re-trying indefinitely

* Deduplication
**  mechanism is obscure/undocumented. Some rely on rootJob parameters, some use
custom logic.
** Batching prevents deduplication. When and how should jobs do batch
operations? Can we automatically break up small batches?
** Delaying jobs may improve deduplication, but support for delayed jobs is
limited/obscure.
** Custom coalescing could improve the chance for deduplication.

* Scope and purpose of some jobs is unclear. E.g. UpdateHtmlCache invalidates
the parser cache, and RefreshLinks re-parse the page - but does not trigger an
UpdateHtmlCache, which it probably should.

* The throttling mechanism does not take into account the nature and run-time of
different job types.

* Scaling is achieved by running more cron jobs.

* Kafka-based JQ is being tested by Services. Generally saner. Should improve
ability to track causality (which job got triggered by which other job). T157088

* No support for recurrent jobs. Should we keep using cron?



-- 
Daniel Kinzler
Principal Platform Engineer

Wikimedia Deutschland
Gesellschaft zur Förderung Freien Wissens e.V.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Should we plan anything regarding Mozilla’s "Talk" open-source commenting platform

[mathieu stumpf guntz]

Hi,

I'm just discovering this blog post Mozilla and the Washington Post Are 
Reinventing Online Comments 
 
and thought it might worth talking about its relevancy for the Wikimedia 
movement. For those who didn't followed the 2017 Movement strategy 
, 
we now also have wikicomment  as a 
solution related to this topic.


Kind regards,
mathieu

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Today: Weekly Technical Advice IRC Meeting

[Michael Schönitzer]

Sorry for cross-posting!

Reminder: Technical Advice IRC meeting again **today 3-4 pm UTC** on
#wikimedia-tech.

The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.

If you know already what you would like to discuss or ask, please add your
topic to the next meeting: https://www.mediawiki.org/
wiki/Technical_Advice_IRC_Meeting

This meeting is an offer by WMDE’s tech team. Hosts of todays meeting are:
@addshore & @Tobi_WMDE_SW.


Hope to see you there!
Michi (for WMDE’s tech team)

-- 
Michael F. Schönitzer



Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Shibboleth authentication

[Sorin Gheorghiu]

Dear developers,

we learned that the authentication layer got a major overhaul in 1.27. 
We are interested in Shibboleth-authentication [1] which derives from 
AuthPlugin.php, deprecated and superseded by AuthManager.php. The last 
version of this extension [1] is compatible with 1.26, but it does work 
with 1.27 and above. It seems the project development has been stopped.


While you are accustomed with the new AuthManager classes, could you 
help us by chance to get a new script working for 1.27? We found the 
updating tips [3] but unfortunately we don't have any Mediawiki 
programming experience.


[1] https://www.mediawiki.org/wiki/Extension:Shibboleth_Authentication
[2] 
https://github.com/ConsortiumGARR/mediawiki-shibboleth-authentication/blob/master/ShibAuthPlugin.php
[3] 
https://www.mediawiki.org/wiki/Manual:SessionManager_and_AuthManager/Updating_tips


Thank you in advance
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wiki farm configuration

[Manuel Vacelet]

On Wed, Sep 13, 2017 at 1:35 AM, MZMcBride  wrote:

> Bartosz Dziewoński wrote:
> >On 2017-09-11 15:41, Manuel Vacelet wrote:
> >>> It is recommended to use a different DB for each wiki (By setting a
> >>> different $wgDBname
> >>> 
> >>>for each wiki). However if you are limited to a single database, you can
> >>>use a different prefix ($wgDBprefix
> >>>  >)
> >>> to separate the different installs.
> >>
> >> But it's not detailed why it's recommended. I'd like to know if there is
> >> any downside of using the prefix strategy (peformances, upgrades, etc) ?
> >
> >I think the only downside is that you'll have to keep the prefixes in
> >mind when writing your own SQL queries. Making and restoring database
> >backups for individual wikis will also be less convenient.
>
> Speaking generally, if there's a MediaWiki-related recommendation, it's
> probably based on the behavior of Wikimedia wikis such as Wikipedia.
> Wikimedia wikis split into one MediaWiki installation per database, though
> many smaller wikis share the same database host. Also, speaking generally,
> it's often easier to combine things that were separate than to separate
> things that were combined.
>
> For your particular question, I think the main considerations are how big
> your wiki farm is in total and how large each wiki is expected to be. In
> my mind, there's a pretty substantial difference between a wiki farm that
> has five members versus a wiki farm that has 800 members. Every MediaWiki
> installation consists of dozens of database tables. And, as alluded to,
> it's typically easier to keep databases on the same host, unless you want
> to go down the road of clustering and sharding. If you're expecting to
> have a wiki farm with five members, but one member will have 5 million
> articles and 4 members will have 200 articles combined, you may want to
> split based on that.
>
> As Bartosz notes, there shouldn't be any issues with using $wgDBprefix
> other than mild inconvenience if you write a lot of database queries
> directly, which is unlikely. $wgDBprefix may be your best option if you
> can only have a single database. (Though maybe, if you're limited to a
> single database, you want to consider a different hosting provider.)
> Bartosz is also correct that lots of tools work best at the
> database-level, though many of them also have support for the
> use-case/setup that you're describing.
>
> Manuel Vacelet wrote:
> >I've got questions around Wiki Farm setup, I'm not sure if it's the right
> >place to ask the question. If there is a better channel, feel free to say
> >so.
>
> This mailing list is a fine place. There's also
> mediawi...@lists.wikimedia.org if you prefer mailing lists, or many IRC
> channels on the freenode network. Potentially helpful channels are
> #mediawiki or #wikimedia-tech.


Thanks to both of you for your detailed answers.

Manuel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] MinusX - a new tool to make sure files aren't executable (+x)

[Legoktm]

Hi,

Occasionally someone will unintentionally create a new file or modify an
existing one to set the executable bit (+x). This is almost nearly an
accident, and usually someone will come along later and fix them en
masse[1][2].

With help from Anomie, I've written a tool, MinusX[3], that will search
and for executable files that shouldn't be, and optionally fix them.
It's written in PHP and should run as part of "composer test", but
operates on all types of files, not just PHP.

I've proposed adding this tool to all repositories[4] in an automated
manner.

If you have any suggestions/feature requests/bugs, feel free to create a
ticket in Phabricator or reply here.

[1] https://phabricator.wikimedia.org/T168659
[2] https://phabricator.wikimedia.org/P5913
[3] https://www.mediawiki.org/wiki/MinusX
[4] https://phabricator.wikimedia.org/T175794

Thanks,
-- Legoktm

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l