Re: [Wikitech-l] Refactoring MonoBook

[Isarra Yos]

On 18/03/18 03:09, K. Peachey wrote:

Just make it MonoBookV5? *runs*, or more realistically V2 then
progressively get rid of the orginal MonoBook?


The problem with splitting things up like that is then you need to deal 
with transitions from a v1 to a v2, which make everyone's lives more 
difficult, and especially create more work for and confuse reusers. As 
long as we can maintain a relatively smooth transition within the 
original, sticking to that is much more ideal - which is why, while I've 
gone and totally refactored MonoBookTemplate, I've also endeavoured to 
ensure the output remains functionally the same, and that all hooks and 
whatnot also continue to be supported, as dumb as several of them are. 
(Right now we're still trying to verify it really IS the same, though, 
so more eyes on that would be appreciated.)


The only other blocker I know of at this point is the other skins that 
straight-up extend MonoBookTemplate, a practice that made rather more 
sense back when this was all part of core, but as there only appear to 
be two of them, it should be pretty straight forward to simply change 
that behaviour now. (Which I've already gone ahead and done for Modern; 
Cavendish has apparently also had a task for doing the same for some 
time, and so shouldn't be far behind.)


-I

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Incoming and outgoing links enquiry

[Erik Bernhardson]

This information is available mostly pre-calculated in the CirrusSearch
dumps at http://dumps.wikimedia.your.org/other/cirrussearch/current/

Each article is represented by a line of json in those dumps. There is a
field called 'incoming_links' which is the number of unique articles with
links from the content namespace(s) to that article. Each article
additionally contains an `outgoing_link` field which contains a list of
strings representing the pages the article links to (incoming_links is
calculated by querying the outgoing_link field). I've done graph work on
wikipedia before using this and the outgoing_link field is typically enough
to build a full graph.



On Sun, Mar 18, 2018 at 2:18 PM, John  wrote:

> I would second the recommendation of using the dumps for such a large
> graphing project. If it's more than a couple hundred pages the API/database
> queries can get bulky
>
> On Sun, Mar 18, 2018 at 5:07 PM Brian Wolff  wrote:
>
> > Hi,
> >
> > You can run longer queries by getting access to toolforge (
> > https://wikitech.wikimedia.org/wiki/Portal:Toolforge) and running from
> the
> > command line.
> >
> > However the query in question might  still take an excessively long time
> > (if you are doing all of wikipedia). I would expect that query to result
> in
> > about 150mb of data and maybe take days to complete.
> >
> > You can also break it down into parts by adding WHERE page_title >='a'
> AND
> > page_title < 'b'
> >
> > Note, also of interest: full dumps of all the links is available at
> >
> > https://dumps.wikimedia.org/enwiki/20180301/enwiki-
> 20180301-pagelinks.sql.gz
> > (you would also need
> > https://dumps.wikimedia.org/enwiki/20180301/enwiki-20180301-page.sql.gz
> to
> > convert page ids to page names)
> > --
> > Brian
> > On Sunday, March 18, 2018, Nick Bell  wrote:
> > > Hi there,
> > >
> > > I'm a final year Mathematics student at the University of Bristol, and
> > I'm
> > > studying Wikipedia as a graph for my project.
> > >
> > > I'd like to get data regarding the number of outgoing links on each
> page,
> > > and the number of pages with links to each page. I have already
> > > inquired about this with the Analytics Team mailing list, who gave me a
> > few
> > > suggestions.
> > >
> > > One of these was to run the code at this link
> > https://quarry.wmflabs.org/
> > > query/25400
> > > with these instructions:
> > >
> > > "You will have to fork it and remove the "LIMIT 10" to get it to run on
> > > all the English Wikipedia articles. It may take too long or produce
> > > too much data, in which case please ask on this list for someone who
> > > can run it for you."
> > >
> > > I ran the code as instructed, but the query was killed as it took
> longer
> > > than 30 minutes to run. I asked if anyone on the mailing list could run
> > it
> > > for me, but no one replied saying they could. The guy who wrote the
> code
> > > suggested I try this mailing list to see if anyone can help.
> > >
> > > I'm a beginner in programming and coding etc., so any and all help you
> > can
> > > give me would be greatly appreciated.
> > >
> > > Many thanks,
> > > Nick Bell
> > > University of Bristol
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] changes coming to large dumps

[Ariel Glenn WMF]

A reprieve!  Code's not ready and I need to do some timing tests, so the
March 20th run will do the standard recombining.

For updates, don't forget to check the Phab ticket!
https://phabricator.wikimedia.org/T179059

On Mon, Mar 5, 2018 at 1:10 PM, Ariel Glenn WMF  wrote:

> Please forward wherever you think appropriate.
>
> For some time we have provided multiple numbered pages-articles bz2 file
> for large wikis, as well as a single file with all of the contents combined
> into one.  This is consuming enough time for Wikidata that it is no longer
> sustainable.  For wikis where the sizes of these files to recombine is "too
> large", we will skip this recombine step. This means that downloader
> scripts relying on this file will need to check its existence, and if it's
> not there, fall back to downloading the multiple numbered files.
>
> I expect to get this done and deployed by the March 20th dumps run.  You
> can follow along here: https://phabricator.wikimedia.org/T179059
>
> Thanks!
>
> Ariel
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What ways are there to include user-edited JavaScript in a wiki page? (threat model: crypto miners)

[David Gerard]

This is particularly important for non-Wikimedia instances of
MediaWiki, by the way.

(e.g. on RationalWiki there's a cultural thing of "everyone is a
sysop!" but the interface/JS editing rights are restricted to a much
smaller "tech" group who are trusted not to be silly)


- d.



On 19 March 2018 at 08:51, Derk-Jan Hartman
 wrote:
> On a side note. Have we looked recently at decoupling the site wide
> JS/CSS rights from the edit interface right ? It has always seemed a
> bit weird to me that we had both these things in MediaWiki namespace,
> but the more we are closing down raw HTML in MediaWiki namespace, the
> weirder it becomes. Even if we would assign those rights to mostly the
> same groups, it would give some more healthy options in the long term.
> We've made a similar split in the user namespace (for much more
> practical reasons of course). But I think that shouldn't stop us from
> doing the same for global stuff.. We could even consider finding some
> way to detect raw html messages and have them subject to the same
> right..
>
> We have https://phabricator.wikimedia.org/T120886 but i'm not sure if
> anyone gave it any serious consideration in the past 2,5 years..
>
> DJ
>
>
> On Sat, Mar 17, 2018 at 6:57 PM, Alex Monk  wrote:
>> You'd have to stop stewards from loading site-wide JS, gadgets, as well as
>> removing their ability to have their user JS from pulling in JS from other
>> sites/users/etc. somehow.
>>
>> Trying to restrict it would probably lead to a backlash from communities
>> that would make superprotect look like a joke. I suspect that if such a
>> feature were proposed today, it would never be given to local users, but
>> reserved for globally trusted people like developers. Local sysops are not
>> necessarily (or maybe even usually) technically skilled, and communities do
>> not appear to realise the amount of power that editinterface actually gives
>> you, and that code written with it may frequently be executed by people
>> with rights that the community would consider superior, like
>> steward/oversight/checkuser/bureaucrat.
>>
>> I would not tell them not to worry about it.
>>
>> On Fri, 16 Mar 2018, 17:33 Leon Ziemba,  wrote:
>>
>>> Sorry to slightly sidetrack this discussion, but someone recently asked me
>>> if it were possible to modify a steward's user JS so that it granted them
>>> advanced rights like steward/checkuser/oversight. This of course is
>>> possible, but very rare since you need to be a sysop to edit these JS
>>> pages. The point this person was making to me however was that on smaller
>>> wikis it can be easy to become a sysop, and it's probable that by nature
>>> stewards will show up there occasionally, and that their own personal JS
>>> may not be closely watched. I told them not to worry about it, but if we
>>> really wanted to do something, we could make a steward's JS only be mutable
>>> by other stewards (or something).
>>>
>>> Maybe something else to think about?
>>>
>>> ~Leon
>>>
>>> On Thu, Mar 15, 2018 at 5:46 PM, Eran Rosenthal 
>>> wrote:
>>>
>>> > Lego already did a script to verify no external resources are loaded:
>>> > https://phabricator.wikimedia.org/T71519
>>> > I think there is a Jenkins job running it on regular basis
>>> >
>>> > On Thu, Mar 15, 2018 at 6:30 AM, MZMcBride  wrote:
>>> >
>>> > > David Gerard wrote:
>>> > > >What ways are there to include user-edited JavaScript in a wiki page?
>>> > > >
>>> > > >[...]
>>> > > >
>>> > > >You can't see it now, but it was someone including a JavaScript
>>> > > >cryptocurrency miner in common.js!
>>> > > >
>>> > > >Obviously this is not going to be a common thing, and common.js is
>>> > > >closely watched. (The above edit was reverted in 7 minutes, and the
>>> > > >user banned.)
>>> > > >
>>> > > >But what are the ways to get user-edited JavaScript running on a
>>> > > >MediaWiki, outside one's own personal usage? And what permissions are
>>> > > >needed? I ask with threats like this in mind.
>>> > >
>>> > > There's an old post of mine that documents some of the ways to inject
>>> > > site-wide JavaScript:
>>> > > >> > August/073787.html
>>> > > >
>>> > >
>>> > > I believe, as Brian notes in this thread, that most methods require
>>> > having
>>> > > the "editinterface" user right so that you can edit wiki pages in the
>>> > > "MediaWiki" namespace. By default, this user right is assigned to the
>>> > > "sysop" user group, but if you search through
>>> > >  for the
>>> > string
>>> > > "editinterface", you can see that on specific wikis such as fawiki,
>>> this
>>> > > user right has been assigned to additional user groups.
>>> > >
>>> > > Jon Robson wrote:
>>> > > >It has always made me a little uneasy that there are wiki pages where
>>> > > >JavaScript could 

Re: [Wikitech-l] What ways are there to include user-edited JavaScript in a wiki page? (threat model: crypto miners)

[Derk-Jan Hartman]

On a side note. Have we looked recently at decoupling the site wide
JS/CSS rights from the edit interface right ? It has always seemed a
bit weird to me that we had both these things in MediaWiki namespace,
but the more we are closing down raw HTML in MediaWiki namespace, the
weirder it becomes. Even if we would assign those rights to mostly the
same groups, it would give some more healthy options in the long term.
We've made a similar split in the user namespace (for much more
practical reasons of course). But I think that shouldn't stop us from
doing the same for global stuff.. We could even consider finding some
way to detect raw html messages and have them subject to the same
right..

We have https://phabricator.wikimedia.org/T120886 but i'm not sure if
anyone gave it any serious consideration in the past 2,5 years..

DJ


On Sat, Mar 17, 2018 at 6:57 PM, Alex Monk  wrote:
> You'd have to stop stewards from loading site-wide JS, gadgets, as well as
> removing their ability to have their user JS from pulling in JS from other
> sites/users/etc. somehow.
>
> Trying to restrict it would probably lead to a backlash from communities
> that would make superprotect look like a joke. I suspect that if such a
> feature were proposed today, it would never be given to local users, but
> reserved for globally trusted people like developers. Local sysops are not
> necessarily (or maybe even usually) technically skilled, and communities do
> not appear to realise the amount of power that editinterface actually gives
> you, and that code written with it may frequently be executed by people
> with rights that the community would consider superior, like
> steward/oversight/checkuser/bureaucrat.
>
> I would not tell them not to worry about it.
>
> On Fri, 16 Mar 2018, 17:33 Leon Ziemba,  wrote:
>
>> Sorry to slightly sidetrack this discussion, but someone recently asked me
>> if it were possible to modify a steward's user JS so that it granted them
>> advanced rights like steward/checkuser/oversight. This of course is
>> possible, but very rare since you need to be a sysop to edit these JS
>> pages. The point this person was making to me however was that on smaller
>> wikis it can be easy to become a sysop, and it's probable that by nature
>> stewards will show up there occasionally, and that their own personal JS
>> may not be closely watched. I told them not to worry about it, but if we
>> really wanted to do something, we could make a steward's JS only be mutable
>> by other stewards (or something).
>>
>> Maybe something else to think about?
>>
>> ~Leon
>>
>> On Thu, Mar 15, 2018 at 5:46 PM, Eran Rosenthal 
>> wrote:
>>
>> > Lego already did a script to verify no external resources are loaded:
>> > https://phabricator.wikimedia.org/T71519
>> > I think there is a Jenkins job running it on regular basis
>> >
>> > On Thu, Mar 15, 2018 at 6:30 AM, MZMcBride  wrote:
>> >
>> > > David Gerard wrote:
>> > > >What ways are there to include user-edited JavaScript in a wiki page?
>> > > >
>> > > >[...]
>> > > >
>> > > >You can't see it now, but it was someone including a JavaScript
>> > > >cryptocurrency miner in common.js!
>> > > >
>> > > >Obviously this is not going to be a common thing, and common.js is
>> > > >closely watched. (The above edit was reverted in 7 minutes, and the
>> > > >user banned.)
>> > > >
>> > > >But what are the ways to get user-edited JavaScript running on a
>> > > >MediaWiki, outside one's own personal usage? And what permissions are
>> > > >needed? I ask with threats like this in mind.
>> > >
>> > > There's an old post of mine that documents some of the ways to inject
>> > > site-wide JavaScript:
>> > > > > August/073787.html
>> > > >
>> > >
>> > > I believe, as Brian notes in this thread, that most methods require
>> > having
>> > > the "editinterface" user right so that you can edit wiki pages in the
>> > > "MediaWiki" namespace. By default, this user right is assigned to the
>> > > "sysop" user group, but if you search through
>> > >  for the
>> > string
>> > > "editinterface", you can see that on specific wikis such as fawiki,
>> this
>> > > user right has been assigned to additional user groups.
>> > >
>> > > Jon Robson wrote:
>> > > >It has always made me a little uneasy that there are wiki pages where
>> > > >JavaScript could potentially be injected into my page without my
>> > approval.
>> > > >To be honest if I had the option I would disable all site and user
>> > scripts
>> > > >for my account.
>> > >
>> > > You could file a Phabricator task about this. We already specifically
>> > > exempt certain pages, such as Special:UserLogin and
>> Special:Preferences,
>> > > from injecting custom JavaScript. We could potentially add a user
>> > > preference to do what you're suggesting.
>> > >
>> > >