[Wikitech-l] Re: Parsing a page: any sections missing

[C. Scott Ananian]

There's a patch in progress for T332243:
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/901690
 --scott

On Thu, Mar 23, 2023 at 2:42 PM JJMC89  wrote:

> Probably February 23 for dewiki. T332243
>  is the bug report due to this
> unannounced breaking API change.
>
>
> On Thu, Mar 23, 2023 at 11:39 AM Martin Domdey  wrote:
>
>> Hi Thiemo,
>>
>> since when is it so? February 22nd it worked properly last time.
>>
>> Kind regards
>> Doc Taxon ...
>>
>> Am Do., 23. März 2023 um 19:35 Uhr schrieb Thiemo Kreuz <
>> thiemo.kr...@wikimedia.de>:
>>
>>> The page contains __NOTOC__. That's why the "sections" property is empty.
>>>
>>> Best
>>> Thiemo
>>> ___
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Wikitext, Document Models, and HTML5 Output

[C. Scott Ananian]

First:  tags are definitely on the near-future roadmap.  There are
some issues with balancing tags (when the section deliberately has an
unclosed ) that are awaiting the completion of the parsoid transition,
but it will certainly happen.

WRT adding additional attributes/properties to certain constructs -- yes,
this is a somewhat pervasive issue with wikitext.  List items and headings
are probably the biggest examples of "un-annotatable" constructs.
Typically folks hack around the issue by using literal  or  tags
in their markup ... but then that leads to the nesting issues described in
the second sentence above.

https://phabricator.wikimedia.org/T230658 contains a discussion, originally
in the context of list item attributes, and in
https://phabricator.wikimedia.org/T230658#5916798 it is proposed to have a
general-purpose `{{#attr}}` parser function which would attach itself to
the nearest containing HTML tag.  More discussion of that proposal in
https://phabricator.wikimedia.org/T230658#5786980.  I think that would
address your use case?
 --scott

On Tue, Jan 11, 2022 at 6:38 PM Adam Sobieski 
wrote:

> Wikitech-l,
>
>
>
> Hello. I have a question about the HTML output of wiki parsers. I wonder
> about how simple or complex that it would be for a wiki parser to output,
> instead of a flat document structure inside of a  element, an
>  element containing nested  elements?
>
>
>
> Recently, in the Community Wishlist Survey Sandbox
> , the
> speech synthesis of Wikipedia articles
> 
> was broached. The proposer of these ideas indicated that, for best results,
> some content, e.g., “See also” sections, should not be synthesized.
>
>
>
> In response to these interesting ideas, I mentioned some ideas from EPUB, 
> referencing
> pronunciation lexicons from HTML
>  and SSML
> attributes in HTML
> ,
> the CSS Speech Module , and that
> output HTML content could be styled using the CSS Speech Module’s speak
> property.
>
>
>
> In these regards, I started thinking about how one might extend wikitext
> syntax to be able to style sections, e.g.,:
>
>
>
> == See also == {style="speak:never"}
>
>
>
> Next, I inspected the HTML of some Wikipedia articles and realized that,
> due to the structure of the output HTML documents, it isn’t simple to style
> or to add attributes to sections. There are only , ,  (et
> cetera) elements inside of a containing  element; sections are not
> yet structured elements.
>
>
>
> The gist is that, instead of outputting HTML like:
>
>
>
> 
>
>   Heading
>
>   Paragraph 1
>
>   Paragraph 2
>
>   Subheading
>
>   Paragraph 3
>
>   Paragraph 4
>
> 
>
>
>
> could a wiki parser output HTML5 like:
>
>
>
> 
>
>   
>
> Heading
>
> Paragraph 1
>
> Paragraph 2
>
> 
>
>   Subheading
>
>   Paragraph 3
>
>   Paragraph 4
>
> 
>
>   
>
> 
>
>
>
> Initial thoughts regarding the latter HTML5 include that it is better
> structured, more semantic, more styleable, and potentially more accessible.
> If there is any interest, I could write up some lengthier discussion about
> one versus the other, why one might be better – and more useful – than the
> other.
>
>
>
> Is this the correct mailing list to discuss any of these wiki technology,
> wiki parsing, wikitext, document model, and HTML5 output topics?
>
>
>
>
>
> Best regards,
>
> Adam
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
(http://cscott.net)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Re: [Wikitech-l]  Wikimedia production errors help

[C. Scott Ananian]

ACN -- for what it's worth, I've been working for the foundation for a
while now, and I can report from the inside that the trend is definitely in
a positive direction.  There is a lot more internal focus on addressing
code debt and giving maintenance a fair spot at the table.  (In fact, my
entire team is now sitting inside 'maintenance' now, apparently; we used to
be 'platform evolution'.)  This email thread is one visible aspect of that
focus on code quality, not just features.

That said, the one aspect which hasn't improved much in my time at the
foundation has been the tendency of teams to work in silos.  This thread
also seems to be a symptom of that: a bunch of production issues are being
dropped on the floor ('not resolved in over a month') because they are
falling between the silos and nobody knows who is best able to fix them.
There are knowledge/expertise gaps among the silos as well: someone
qualified to fix a DB issue might be at sea trying to track down a front
end bug, and vice-versa---a number of generalists in the org could
technically tackle a bug no matter where it lies, but it will take them
much longer to grok an unfamiliar codebase than it would for someone more
familiar with that silo.  So bug triage is an increasingly technical task
in its own right.

This thread, as I read it sitting inside the org, isn't so much asking for
more attention to be paid to maintenance -- we're winning that battle,
internally -- as it is a plea for those folks on the edges of their silos
to keep an eye out for these things which are currently falling between
them and help with the triage.
  --scott, speaking only for myself and my view here



On Wed, Sep 16, 2020 at 11:25 PM AntiCompositeNumber <
anticompositenum...@gmail.com> wrote:

> There is an impression among many community members, myself included,
> that Foundation development generally prioritizes new features over
> fixing existing problems. Foundation teams will sprint for a few
> months to put together a minimum viable product, release it, then move
> on to the new hotness, leaving user requests, bugfixes, and the like
> behind. It often seems that the only way to get a bug fixed is to get
> a volunteer developer to look at it. This is likely unintentional, but
> it happens nonetheless.
>
> Putting a higher priority within the Foundation on cleaning up old
> toys before taking out new ones is necessary for the long-term
> stability of the projects.
>
> ACN
>
> On Wed, Sep 16, 2020 at 9:05 PM Dan Andreescu 
> wrote:
> >
> > >
> > > For example, of the 30 odd backend errors reported in June, 14 were
> still
> > > open a month later in July [1], and 12 were still open – three months
> later
> > > – in September. The majority of these haven't even yet been triaged,
> > > assigned assigned or otherwise acknowledged. And meanwhile we've got
> more
> > > (non-JavaScript) stuff from July, August and September adding
> pressure. We
> > > have to do better.
> > >
> > > -- Timo
> > >
> >
> > This feels like it needs some higher level coordination.  Like perhaps
> > managers getting together and deciding production issues are a priority
> and
> > diverting resources dynamically to address them.  Building an awesome new
> > feature will have a lot less impact if the users are hurting from growing
> > disrepair.  It seems to me like if individual contributors and
> maintainers
> > could have solved this problem, they would have by now.  I'm a little
> > worried that the only viable solution right now seems like heroes
> stepping
> > up to fix these bugs.
> >
> > Concretely, I think expanding something like the Core Platform Team's
> > clinic duty might work.  Does anyone have a very rough idea of the time
> it
> > would take to tackle 293 (wow we went up by a dozen since this thread
> > started) tasks?
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Question to WMF: Backlog on bugs

[C. Scott Ananian]

I'll echo Andre here: the specific problem of patches from new volunteer
devs which don't get timely responses is a real issue, and one which we
have attempted to address (as Andre described) but an area we could
probably still use additional ideas, accountability, etc for.

A secondary issue is that too much wiki dev is done by WMF/WMFDE employees
(IMO); I don't think the current percentages lead to an overall healthy
open source community. But (again in my view) the first step to nurturing
and growing our non-employee contributors is to make sure their patches are
timely reviewed.
  --scott

On Sat, Mar 16, 2019, 10:54 PM Andre Klapper  wrote:

> Hi and thanks for joining the discussion!
>
> On Sat, 2019-03-16 at 20:37 -0400, Thomas Eugene Bishop wrote:
> > Here’s a specific example, created in 2015:
> >
> >   https://phabricator.wikimedia.org/T116145 <
> > https://phabricator.wikimedia.org/T116145>
> >
> >
> > A bug fix was provided years ago but never accepted or rejected. It’s
> > the first and last MediaWiki bug ever assigned to me. I’ve just
> > unassigned myself.
> >
> > In cases like this, remarks like “Because you did not fix these bugs”
> > and “... anyone is free to pick it up and work on it ... No further
> > response needed” miss the point. When a bug fix is provided, but
> > nobody with authority to accept or reject it ever does so, that’s a
> > failure on the part of those who have authority, not on the part of
> > those who are able and willing to fix bugs. Sure, volunteers are
> > “free” to waste their time!
> >
> > You need to use and share your authority more effectively, to “be
> > bold” with accepting and rejecting bug fixes. Authorize more people
> > to accept or reject bug fixes. Assign each proposed bug fix to one
> > such person, starting with the oldest bugs. Then hold those people
> > accountable. You don’t lack volunteers, you lack volunteers with
> > authority.
>
> I fully agree. I was referring to bug reports in my emails.
>
> Code review is an area in which Wikimedia is very frustrating. There
> are regular emails about patches by new contributors awaiting review
> [1] but that obviously only covers a small group of contributors.
> And while we recently started to have code stewardship reviews [2] to
> fill some gaps in the list of responsible persons and teams [3] per
> code base, we for example still lack meaningful statistics how big the
> code review problem is, in general and per team.
>
> andre
>
> [1]
> https://lists.wikimedia.org/pipermail/wikitech-l/2019-March/091632.html
> [2] https://www.mediawiki.org/wiki/Code_stewardship_reviews
> [3] https://www.mediawiki.org/wiki/Developers/Maintainers
> --
> Andre Klapper | Bugwrangler / Developer Advocate
> https://blogs.gnome.org/aklapper/
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] $wgVisualEditorParsoidForwardCookies

[C. Scott Ananian]

That's a good question!  I'm not aware of any recent changes that would
have made $wgVisualEditorParsoidForwardCookies not necessary, but I *do*
recall that we don't use those long-form configuration variable names any
more.

Latest documentation seems to be at
https://www.mediawiki.org/wiki/Extension:VisualEditor#Linking_with_Parsoid_in_private_wikis

Hope this helps!
 --scott

On Fri, Oct 19, 2018 at 7:25 AM Aran via Wikitech-l <
wikitech-l@lists.wikimedia.org> wrote:

> Hi,
>
> just wondering what the situation is with
> $wgVisualEditorParsoidForwardCookies these days. The documentation for
> visual editor says that it's needed if you have a wiki that's not
> editable by the public. But I've just set it up on such a wiki and
> editing/saving an article through visual editor works fine without me
> having set this or allowed anonymous edits by localhost. Is there no
> longer any need to worry about settings specifically for locked down wikis?
>
> Thanks,
>
> Aran
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Collecting UI feedback for PolyGerrit - Gerrit

[C. Scott Ananian]

Just as a wishlist item, I wonder if we could use something like
https://tools.ietf.org/html/rfc1760 to convert gerrit #s (and phab #s) to
short word strings.  I find myself typing sequences of six+ digits over and
over again during the workday, and the difficulty of getting these exactly
right (esp transpositions) makes me resort to copy-paste etc.  And of
course in oral meetings reciting these digit strings is especially fun.  It
would be much nicer to say `git review -d "top huh"` rather than `git
review -d 467470`...
  --scott

On Tue, Oct 16, 2018 at 11:34 AM Paladox via Wikitech-l <
wikitech-l@lists.wikimedia.org> wrote:

>  I must note that with the branching of 2.16, it means that GWTUI is going
> away. With the removal of GWTUI in
> https://gerrit-review.googlesource.com/c/gerrit/+/116790
>
> On Tuesday, 16 October 2018, 16:30:53 BST, Paladox via Wikitech-l <
> wikitech-l@lists.wikimedia.org> wrote:
>
>   Upstream have just branched 2.16 with a rc release a few hours away! So
> a stable release should be done shortly with all the improvements to
> polygerrit's ui and a inline editor too!
> On Friday, 5 October 2018, 19:26:30 BST, Paladox via Wikitech-l <
> wikitech-l@lists.wikimedia.org> wrote:
>
>   I have filed it upstream at
> https://bugs.chromium.org/p/gerrit/issues/detail?id=9815
> On Friday, 5 October 2018, 06:54:25 BST, Dalba 
> wrote:
>
>  I have issues with copying the text of changed files:
>
> `ctrl+a` does not work as it used to anymore: While using the
> side-by-side diff view, go to a changed file, click on the old/new
> revision, press `ctrl+a`. In the old UI only the text of the selected
> file would have been selected, but now the whole page gets selected
> which is not that useful.
>
> Also, copying multiple lines results in extra whitespace being copied
> in the middle of the lines. For example goto [1], select the text of
> both lines and copy... It'll be copied as `# -*- coding: utf-8
> -*-\n\t\n\t\n"""Package to hold all library tests."""` (note the
> `\n\t\n\t\n` which should have been just a `\n`).
>
> [1]:
> https://gerrit.wikimedia.org/r/c/pywikibot/core/+/463061/5/tests/library_tests/__init__.py#1
>
> On Thu, Oct 4, 2018 at 12:39 AM Paladox via Wikitech-l
>  wrote:
> >
> > Hi, i am collecting feedback for Gerrit's New UI called PolyGerrit. It's
> possible to use PolyGerrit on gerrit.wikimedia.org since 2.14. The new UI
> has recently been made the default upstream. The Old UI is going away in
> the next release after 2.16. Upstream have given PolyGerrit another update
> that looks different to the one on gerrit.wikimedia.org. PolyGerrit now
> includes a dark ui.
> >
> > To switch to PolyGerrit either click the "New UI" button on the footer
> or put ?polygerrit=1 in the url.
> >
> > To switch back to GWTUI either click "Switch back to old ui" on the
> footer or put ?polygerrit=0 in the url.
> >
> > Non dark mode:
> >
> > Here's how it looks like:
> >
> > Dashboard:
> >
> > https://phabricator.wikimedia.org/F26296230
> >
> >
> > Change list:
> >
> > https://phabricator.wikimedia.org/F26296240
> >
> > Change screen:
> >
> > https://phabricator.wikimedia.org/F26296242
> >
> >
> > https://phabricator.wikimedia.org/F26296257
> >
> >
> > Dark mode: https://phabricator.wikimedia.org/F26296282
> >
> >
> > And many other UI improvements across the app.
> >
> > You can play around the the new ui from the master branch that will
> become 2.16 here https://gerrit.git.wmflabs.org/r/
> >
> > Please give feedback so upstream can make PolyGerrit even better! You
> can either file your reports at
> https://phabricator.wikimedia.org/project/view/330/ or reply to the email
> with your feedback.
> >
> >
> >
> >
> > It has a dedicated team on the UI with a design researcher behind the
> scenes redesigning polygerrit constantly based on feedback.
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Breaking change: return value for OutputPage::adaptCdnTTL() removed

[C. Scott Ananian]

There are no known users of the return value of OutputPage::adaptCdnTTL()
according to
https://codesearch.wmflabs.org/search/?q=adaptCdnTTL=nope==

But our all-seeing eye does not see everything.  Be it known that
https://gerrit.wikimedia.org/r/450075, just merged, removed the return
value for OutputPage::adaptCdnTTL() and this is technically a breaking
change.

The value formerly returned by adaptCdnTTL() was arguably misleading:
callers would probably expect it to return mCdnMaxage, but instead it
(formerly) returned mCdnMaxageLimit.  Since the return value was easily
misunderstoood *and* it was unused, we've decided to remove it.

@simetrical (Aryeh) wrote the patch and should get the credit (the test
coverage for OutputPage was also significantly increased! Yay, Aryeh!).
Krinkle and I thought it was worth a C+2, so if in fact
OutputPage::adaptCdnTTL() was vital to your way of life and we've ruined it
forever, we should get the blame.
  --scott
-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] non-obvious uses of in your language

[C. Scott Ananian]

The relevant Parsoid feature request for having VE use linktrails is
https://phabricator.wikimedia.org/T50463 since in general Parsoid just
generates [[Book|books]] when VE gives it `books`.

If VE gives Parsoid `books` it will assume that's what
the author actually meant, and will generate `[[Book]]s` using a
very general mechanism used for a number of other syntax conflicts (like if
you actually want to start a line with the literal character `*`).

I don't think the answer is to invent new syntax for linktrail separation
-- we already have quite enough different ways of escaping and/or
token-breaking already, as partially enumerated in this thread already.
The only one I would be happy to faciliatate would be `-{}-` since it is
already an odd parser corner case -- it is parsed by the wikitext
preprocessor but then spit back out as literal text by the second parsing
phase unless LanguageConverter is enabled for the specific page language.
It would simplify the parse if the LanguageConverter constructs were
"always on" instead of being en/disabled on a page-by-page basis.
  --scott

On Sun, Oct 7, 2018 at 12:23 PM Amir E. Aharoni <
amir.ahar...@mail.huji.ac.il> wrote:

> ... And, more importantly, its form doesn't say "separate the trail from
> the link". Just like , it only *happened* to do it (I tried on
> Wikipedia, and it doesn't do it now).
>
> The point I'm trying to make in this thread is that  happens to do
> certain things other than showing wiki syntax without parsing, and is used
> for them as if it's *intended* for it, but this is a hack. If a certain
> functionality is needed, such as separating the trail from the link, then
> it's worth considering creating a piece of syntax for it.
>
> --
> Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
> http://aharoni.wordpress.com
> ‪“We're living in pieces,
> I want to live in peace.” – T. Moore‬
>
>
> ‫בתאריך יום א׳, 7 באוק׳ 2018 ב-19:08 מאת ‪bawolff‬‏ <‪bawolff...@gmail.com
> ‬‏>:‬
>
> > Alas, no longer valid in XML or HTML5. (Although HTML5 will still
> > parse it as an empty comment, but with a  "incorrectly-opened-comment"
> > error.
> >
> > --
> > Brian
> >
> >
> > On Sat, Oct 6, 2018 at 6:57 AM Chad  wrote:
> > >
> > > Found it :)
> > >
> > > https://www.w3.org/MarkUp/SGML/sgml-lex/sgml-lex
> > >
> > > Search for "empty comment declaration" :)
> > >
> > > -Chad
> > >
> > > On Fri, Oct 5, 2018, 11:50 PM Chad  wrote:
> > >
> > > > I'm personally a fan of .
> > > >
> > > > I came across it years ago--it's a null comment. Can't find the
> > reference
> > > > at the moment though.
> > > >
> > > > -Chad
> > > >
> > > > On Thu, Oct 4, 2018, 2:25 PM Daniel Kinzler 
> > > > wrote:
> > > >
> > > >> Am 04.10.2018 um 18:58 schrieb Thiemo Kreuz:
> > > >> > The syntax "[[Schnee]]reichtum" is quite common in the
> > > >> > German community. There are not many other ways to achieve the
> same:
> > > >> >  or  can be used instead.[1] The later is often the
> > > >> > better alternative, but an auto-replacement is not possible. For
> > > >> > example, "[[Bund]]estag" must become
> "[[Bund]]estag".
> > > >>
> > > >> We could introduce new syntax for this, such as  or even
> > .
> > > >>
> > > >> Or how about {{}} for "this is a syntactic element, but it does
> > nothing"?
> > > >> But if
> > > >> that is mixed in with template expansion, it won't work if it
> expands
> > to
> > > >> nothing, since template expansion happens before link parsing,
> right?
> > For
> > > >> better
> > > >> or worse...
> > > >>
> > > >> --
> > > >> Daniel Kinzler
> > > >> Principal Software Engineer, MediaWiki Platform
> > > >> Wikimedia Foundation
> > > >>
> > > >> ___
> > > >> Wikitech-l mailing list
> > > >> Wikitech-l@lists.wikimedia.org
> > > >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > >
> > > >
> > > On Oct 5, 2018 11:50 PM, "Chad"  wrote:
> > >
> > > I'm personally a fan of .
> > >
> > > I came across it years ago--it's a null comment. Can't find the
> reference
> > > at the moment though.
> > >
> > > -Chad
> > >
> > > On Thu, Oct 4, 2018, 2:25 PM Daniel Kinzler 
> > wrote:
> > >
> > > > Am 04.10.2018 um 18:58 schrieb Thiemo Kreuz:
> > > > > The syntax "[[Schnee]]reichtum" is quite common in the
> > > > > German community. There are not many other ways to achieve the
> same:
> > > > >  or  can be used instead.[1] The later is often the
> > > > > better alternative, but an auto-replacement is not possible. For
> > > > > example, "[[Bund]]estag" must become
> "[[Bund]]estag".
> > > >
> > > > We could introduce new syntax for this, such as  or even
> > .
> > > >
> > > > Or how about {{}} for "this is a syntactic element, but it does
> > nothing"?
> > > > But if
> > > > that is mixed in with template expansion, it won't work if it expands
> > to
> > > > nothing, since template expansion happens before link parsing, right?
> > For
> > > > better
> > > > or worse...
> > > >
> > > > --
> > > > Daniel Kinzler
> > > > 

Re: [Wikitech-l] non-obvious uses of in your language

[C. Scott Ananian]

-{}- is already commonly used on LanguageConverter wikis for "this is a
syntactic element but does nothing except separate a word".
The preprocessor already understands it on all wikis, as well.  (But then
we explicitly serialize it to literally `-{}-` if your content language
doesn't have variants defined.)
  --scott

On Thu, Oct 4, 2018 at 5:25 PM Daniel Kinzler 
wrote:

> Am 04.10.2018 um 18:58 schrieb Thiemo Kreuz:
> > The syntax "[[Schnee]]reichtum" is quite common in the
> > German community. There are not many other ways to achieve the same:
> >  or  can be used instead.[1] The later is often the
> > better alternative, but an auto-replacement is not possible. For
> > example, "[[Bund]]estag" must become "[[Bund]]estag".
>
> We could introduce new syntax for this, such as  or even .
>
> Or how about {{}} for "this is a syntactic element, but it does nothing"?
> But if
> that is mixed in with template expansion, it won't work if it expands to
> nothing, since template expansion happens before link parsing, right? For
> better
> or worse...
>
> --
> Daniel Kinzler
> Principal Software Engineer, MediaWiki Platform
> Wikimedia Foundation
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Collecting UI feedback for PolyGerrit - Gerrit

[C. Scott Ananian]

The new Gerrit UI seems to be missing the "ignore whitespace" option in
diffs which the old Gerrit UI had.  That's a very useful feature when
reviewing patches which include indentation changes, for example compare:
https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/463559/6/lib/wt2html/TokenTransformManager.js?polygerrit=1
with
https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/463559/6/lib/wt2html/TokenTransformManager.js?polygerrit=0
(after setting "ignore whitespace" to "all" in the gear-shaped preference
menu in the top-right).
 --scott

On Wed, Oct 3, 2018 at 5:09 PM Paladox via Wikitech-l <
wikitech-l@lists.wikimedia.org> wrote:

> Hi, i am collecting feedback for Gerrit's New UI called PolyGerrit. It's
> possible to use PolyGerrit on gerrit.wikimedia.org since 2.14. The new UI
> has recently been made the default upstream. The Old UI is going away in
> the next release after 2.16. Upstream have given PolyGerrit another update
> that looks different to the one on gerrit.wikimedia.org. PolyGerrit now
> includes a dark ui.
>
> To switch to PolyGerrit either click the "New UI" button on the footer or
> put ?polygerrit=1 in the url.
>
> To switch back to GWTUI either click "Switch back to old ui" on the footer
> or put ?polygerrit=0 in the url.
>
> Non dark mode:
>
> Here's how it looks like:
>
> Dashboard:
>
> https://phabricator.wikimedia.org/F26296230
>
>
> Change list:
>
> https://phabricator.wikimedia.org/F26296240
>
> Change screen:
>
> https://phabricator.wikimedia.org/F26296242
>
>
> https://phabricator.wikimedia.org/F26296257
>
>
> Dark mode: https://phabricator.wikimedia.org/F26296282
>
>
> And many other UI improvements across the app.
>
> You can play around the the new ui from the master branch that will become
> 2.16 here https://gerrit.git.wmflabs.org/r/
>
> Please give feedback so upstream can make PolyGerrit even better! You can
> either file your reports at
> https://phabricator.wikimedia.org/project/view/330/ or reply to the email
> with your feedback.
>
>
>
>
> It has a dedicated team on the UI with a design researcher behind the
> scenes redesigning polygerrit constantly based on feedback.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] non-obvious uses of in your language

[C. Scott Ananian]

https://hu.wikipedia.org/w/index.php?title=Grafikus_matroid=prev=20406147
illustrates another use: separating - and { in the unusual case where this
string is wanted and you *don't* want language converter markup.  ie
`-{foo}-` is different from `-{foo}-`.  You don't usually notice
this because languageconversion is disabled in many wikis, but it can cause
problems if unbalanced syntax is used inside a template argument, like:
`{{foo|-{bar}}`.  Here you need to use `{{foo|-{bar}}`, even if
LanguageConverter is not enabled.

Amir -- in german, shouldn't they be tweaking the "linktrail" setting on
dewiki, instead of using ``?  What are cases where they *do* want
the link to include the entire word?  Can they be automatically
distinguished?
 --scott

On Thu, Oct 4, 2018 at 11:17 AM Bináris  wrote:

> Here is a list of removals. :-)
>
> https://hu.wikipedia.org/w/index.php?title=Speci%C3%A1lis:Szerkeszt%C5%91_k%C3%B6zrem%C5%B1k%C3%B6d%C3%A9sei/BinBot=20180912205000=BinBot=28
>
>
> Amir E. Aharoni  ezt írta (időpont: 2018.
> okt. 4., Cs, 16:47):
>
> > Thanks. Can you please give some particular examples?
> >
> > בתאריך יום ה׳, 4 באוק׳ 2018, 17:41, מאת Bináris ‏:
> >
> > > Amir E. Aharoni  ezt írta (időpont:
> 2018.
> > > okt. 4., Cs, 16:18):
> > >
> > > >
> > > > This sentence shows the template used at the end.{{Citation
> > > > needed|reason=Reliable source needed for the whole
> > sentence|date=October
> > > > 2018}}
> > > >
> > > > However,  has less trivial use cases, that are not quite the
> > same
> > > > as demonstrating wiki syntax. One such usage I'm aware of is linking
> a
> > > part
> > > > of a long compound German word, for example "[[Schnee]] > > />reichtum".
> > > > It produces the desired effect, however it is a bit of a hack: the
> word
> > > > "nowiki" doesn't have anything to do with dividing compound words.
> This
> > > use
> > > > is quite common in the German Wikipedia because of the nature of the
> > > German
> > > > language, which has a lot of long compound words.
> > > >
> > >
> > > We have a lot of them in Hungarian Wikipedia, and we have just decided
> to
> > > eradicate them, because this is a non-desired effect. :-)
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Bináris
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Datacenter switchover and switchback

[C. Scott Ananian]

Oct 8 seems to be a particularly bad time to freeze the train given that we
are forking for the MW 1.32 release on Oct 15, and a lot of folks have
last-minute things they want to get into the release (eg deprecations, etc).
  --scott

On Thu, Aug 30, 2018 at 10:57 AM Pine W  wrote:

> +1 to DJ's question about timing. Also, one might wish to be mindful of
> the number of recent trains that were supposed to be boring but involved
> interesting surprises; this makes me wonder whether trains that one thinks
> will be boring are actually OK in this circumstance even if they turn out
> to be "interesting".
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
>
>  Original message From: Derk-Jan Hartman <
> d.j.hartman+wmf...@gmail.com> Date: 8/30/18  2:54 AM  (GMT-08:00) To:
> Wikimedia developers  Subject: Re:
> [Wikitech-l] Datacenter switchover and switchback
> While I think these regular switches are a very good idea, from an outside
> perspective I do have to question a process that puts a significant plug in
> the velocity of various teams working on major projects (esp. in a time of
> year that could probably be seen as one of the most productive). What are
> plans to reduce the disruption of this exercise in the future ?
>
> DJ
>
> On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo 
> wrote:
>
> > Let me explain the rationale of the bellow request for clarification:
> >
> > On Wed, Aug 29, 2018 at 11:30 PM MA  wrote:
> >
> > > Hello:
> > >
> > > >For the duration of the switchover (1 month), deployers are kindly
> > > >requested to refrain from large db schema changes and avoid deploying
> > > >any kind of new feature that requires creation of tables.
> > > >There will be a train freeze in the week of Sept 10th and Oct 8th.
> >
> >
> > During the failover, some schema changes will be finalized on the current
> > active datacenter (plus some major server and network maintenance may be
> > done)- our request is mostly to refrain from quickly enabling those large
> > new unlocked features (e.g. the ongoing comment refactoring, actor/user
> > refactoring, Multi Content Revision, JADE, major wikidata or structured
> > comons structure changes, new extensions not ever deployed to the
> cluster,
> > etc.) at the same time than the ongoing maintenance to reduce variables
> of
> > things that can go bad- enabling those features may be unblocked during
> the
> > switchover time, but we ask you to hold until being back on the current
> > active datacenter. Basically, ask yourself if you are enabling a large
> new
> > core feature or want to start a heavy-write maintenance script and there
> is
> > a chance you will need DBA/system support. Sadly, we had some instances
> of
> > this happening last year and we want to explicitly discourage this during
> > these 2 weeks.
> >
> > In own my opinion, enabling existing features on smaller projects (size
> > here is in amount of server resources, not that they are less important)
> is
> > equivalent to a swat change, and I am not against it happening. I would
> ask
> > contributors to use their best judgement on every case, and ask people on
> > the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt.
> > My plea is to not enable major structural changes during that time may
> > affect thousands of edits per minute. Swat-like changes and "boring" :-)
> > trains are ok.
> >
> > For new wiki creations I would prefer if those were delayed but CC #DBA s
> > on the phabricator task to check with us.
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikitech-ambassadors] Translations on hold until further notice

[C. Scott Ananian]

On Thu, Sep 27, 2018 at 11:51 AM Brad Jorsch (Anomie) 
wrote:

>
> "Added since 2018-09-27" would only be marginally better. Old overrides
> wouldn't be lost, but you'd still wipe out new overrides that are intended
> to actually be overrides rather than workarounds for this issue.
>
>
Would that be worse than turning away missing/incorrect translations for
?

Depends on how long  is, I guess.  I don't think we ever got
an estimate of that?
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikitech-ambassadors] Translations on hold until further notice

[C. Scott Ananian]

On Thu, Sep 27, 2018 at 9:44 AM Brad Jorsch (Anomie) 
wrote:

> On Thu, Sep 27, 2018 at 9:37 AM C. Scott Ananian 
> wrote:
>
> > Couldn't you also make sure to edit things in both places?  That is, make
> > the edit on translatewiki, then manually copy it over to your local wiki,
> > and expect that it will be overwritten when updates from translatewiki
> are
> > turned back on?
> >
>
> The "expect that it will be overwritten" part isn't how it works. The local
> copy will remain in effect until someone deletes it.
>

Wouldn't "delete all local overrides" (or "all local overrides added since
2018-09-27") be a reasonable step to include in migration after the "new
translatewiki" is turned back on?
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikitech-ambassadors] Translations on hold until further notice

[C. Scott Ananian]

On Thu, Sep 27, 2018 at 9:06 AM bawolff  wrote:

> Updates from translatewiki will be put on pause. You can continue to update
> translations at translatewiki, but they won't show up on Wikimedia wikis
> until the current issues are sorted out.
>
> In order to avoid things getting out of sync with translatewiki, we would
> like to ask that you avoid translating things locally unless you really
> need to until updates from translate wiki are turned back on.
>

Couldn't you also make sure to edit things in both places?  That is, make
the edit on translatewiki, then manually copy it over to your local wiki,
and expect that it will be overwritten when updates from translatewiki are
turned back on?
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] My Phabricator account has been disabled

[C. Scott Ananian]

I'd like to echo/reinforce Adam's conclusion:

On Wed, Aug 8, 2018 at 7:39 PM, Adam Wight  wrote:

>
> Thank you for your energy and insights, and I hope we can work together to
> root out the bad decisions and corruption, without this nonsense of having
> to bail you out of Phabricator jail every few months.
>

FWIW, I value the "loyal opposition" in open source projects as a healthy
counterweight.  I have worked with other open source projects in the past
where the "loyal opposition" proved to outlast the original project in
dedication to the shared cause.

I also care deeply about preventing harassment of our community.  It is a
hard line to draw beween "difficult truths" and "deliberately hurtful".
"Assume good faith" is our mantra to try to concentrate on the "truth"
instead of the "difficult" part, but it's a never-ending challenge to get
the balance right.

MZ is a canary in our coal mine, in two ways.  On one hand MZ continually
challenges us to revisit our assumptions and do better in our work.  This
is hard but terribly useful.

On the other hand, MZ tests and probes our community guidelines.  We need
to ensure they are well calibrated to protect the community from harm.  And
I'm not going to minimize the harm that careless criticism can do,
especially to new contributors or soft voices.  I don't think a temporary
ban in this case is outrageous (although I echo Chad's concern), and I
don't think that close scrutiny of MZ's words is unreasonable.  I think
there are many measures we can take to listen carefully to MZ without
allowing MZ's actions to effect harm, and we should continue to do them.

I hope that we will continue to do the difficult balancing work, and not
fall into the easy extremes.  We must not ignore difficult truths, though
it is easy to do so if the messenger is unappealing. But we also must not
assume that because the criticism is legit the presentation is ipso facto
acceptable.
  --scott, speaking for myself only

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] [Ops] MW Train status: Incomplete rollout, blocked on two issues

[C. Scott Ananian]

FWIW, I think the Babel bug was waiting for several days for review from
Niklas, the extension owner, who wasn't at wikimania.  But the folks who
ended up taking action and doing the revert we're at wikimania -- I think
it's a general matter of lack of communication during wikimania, with folks
distracted and assuming someone else is responsible, rather than a specific
"X person is at location Y". Wikimania just disrupts normal communications
and work patterns.
  --scott

(To be clear, I'm not trying to pass blame, I'm just pointing out that the
disruption is not as simple of who is where or whether "enough" people are
at home.)

On Mon, Jul 23, 2018, 4:42 PM Greg Grossmeier  wrote:

> 
> > Deployments during the week of wikimania are always tough.  I'm surprised
> > we even tried to do that this year.
>
> There is/was sufficient coverage from both RelEng and SRE so there
> wasn't an issue of site outage without available people to respond. IOW,
> no one from RelEng nor SRE was attending Wikimania (sadly).
>
> There is the issue of not getting quick feedback and resolution on UBN!
> bugs from attendees at the conference, though. (This was obviously
> dependent on who was at the conference and only manifested itself in a
> couple cases.)
>
> Greg
>
> --
> | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
> | Release Team ManagerA18D 1138 8E47 FAC8 1C7D |
>
> ___
> Engineering mailing list
> engineer...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Ops] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

[C. Scott Ananian]

Deployments during the week of wikimania are always tough.  I'm surprised
we even tried to do that this year.
  --scott

On Mon, Jul 23, 2018, 7:23 PM Željko Filipin  wrote:

> Hi,
>
> This is the second ever train I am conducting and it has been very
> interesting. I've been told that is not unusual. I have good news and bad
> news.
>
> Good news is that some problems are resolved. I would like to thank to
> everybody that helped.
>
> Bad news is that train is still blocked. :( Blocking tasks are:
>
> - T199941 Fatal MWException in Babel: "Language::isValidBuiltInCode must
> be passed a string"
> - T199983 Wikidata showing wrong language for page elements
> - T200136 Does not work for change a log type drop down when the log type
> specified by URL / argument @ 1.32.0-wmf.13 (360f7b5)
>
> I will cut the new branch tomorrow, but I will not be able to deploy it to
> group 0 until blockers from last week are resolved. If you can help
> resolving the current problems, please do. If you know that somebody can
> help, please let them know.
>
> Željko
>
>
> On Fri, Jul 20, 2018 at 7:19 PM Greg Grossmeier 
> wrote:
>
>> Hello,
>>
>> The 1.32.0-wmf.13 version of MediaWiki is rollout to almost all group0
>> and group1 wikis, but not group2. In plain wording: It is deployed to
>> all non-wikipedias (Commons, wiktionaries, etc) except Wikidata.
>>
>> It was blocked from going to all wikis yesterday due to two issues found
>> during the week:
>> * Fatal MWException in Babel: "Language::isValidBuiltInCode must be
>>   passed a string" - https://phabricator.wikimedia.org/T199941
>>
>> * Wikidata showing wrong language for page elements -
>>   https://phabricator.wikimedia.org/T199983
>>
>> Assuming these issues are resolved before Monday we hope to resume the
>> deployment of this version Monday during European working hours.
>>
>> The tracking task for this deployment:
>> https://phabricator.wikimedia.org/T191059
>>
>> A handy tool to see which wikis have which version:
>> https://tools.wmflabs.org/versions/
>>
>> Greg
>>
>> --
>> | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
>> | Release Team ManagerA18D 1138 8E47 FAC8 1C7D |
>> ___
>> Engineering mailing list
>> engineer...@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>
> ___
> Ops mailing list
> o...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] eslint compromised, reset your npm tokens

[C. Scott Ananian]

Further eslint-related packages seem to be infected:
https://github.com/eslint/eslint/issues/10600

All WM devs with publish access to npm should be using 2FA, which would
mitigate this issue.

All WM node packages should also be using npm shrinkwrap files; we should
probably audit that.
 --scott

On Thu, Jul 12, 2018 at 11:30 AM, Kunal Mehta 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi,
>
> If you ran eslint (JavaScript codestyle linter) recently (it was only
> compromised for an hour), your npm token might have been compromised
> (~/.npmrc).
>
> To identify if you were compromised, run:
> $ locate eslint-scope | grep -i "eslint-scope/package.json" | xargs jq
> .version
>
> And if any of those show "3.7.2" then you have the bad package version
> installed.
>
> Upstream recommends that you 1) reset your npm token and 2) enable 2fa
> for npm - both can be done from the npm website. You should probably
> also check to make sure none of your packages were compromised.
>
> There are some more details on the bug report[1].
>
> [1]
> https://github.com/eslint/eslint-scope/issues/39#issuecomment-404533026
>
> - -- Legoktm
> -BEGIN PGP SIGNATURE-
>
> iQIzBAEBCgAdFiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAltHdC0ACgkQUvyOe+23
> /KJpBg//WXBSPKhjmZd43KrHu07NsasWvrU/SAOeBtKjdaLTA3Ry5N+Fdh7LUFFk
> oEm1rnz6AnfW0LPIbiDn66FTJ7jF1X6sV1GxpKhFQyYs6SL7LL4wT/XplRSwUTTD
> hHccwuqPueYpD208w0zRcWVO7wpU7Lm+8xFrVwjhK7Q1AF6GzfwtmHy22fY05doM
> NzXvYgB9urC1fYPQsEO6IhgNH7DT+ZtYOiHnRk2vTgr3fkIjKh4bNEdrnaQ9TOH5
> junlio+07llaF/gB/JWycctuy2z2T/zENLPwhy9ZK35DgikGaMsDU7mA6iGgoxhc
> TQPDnn3Veel7FBXMPCrxYMDgcBCEqENdOfQcbEl9lXDocr7UjQF/0GsvhFncMoIY
> GCfdSThYV6x/U9StyBdxerbX4fCddPgd2RvKjVgDmOdsOVGCU0/iKyhgrBh3AbfP
> MNf+AzYCUGvnzfDsDIF+CvJhcddSHX44N5TGLubVwIMIHsvBevC+7D9uHGaLqkem
> UR8xa489SZ8LOnsL8TgtRaGXNaWqeJX7tIGPtiS5s2bzhRDr8q062VOd3J/Qw3E0
> AQSixX+dQezw282RHYpCk3xuRgbN1oKvCEbOyDB97sbo19f+W2k0CmPVxIaDkr50
> D729WS+6XvozYaw0z/R1aOWJTJLTe9ZUO/Zi9qhDfQtLVzTz8M8=
> =WybD
> -END PGP SIGNATURE-
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Can/should my extensions be deleted from the Wikimedia Git repository?

[C. Scott Ananian]

On Fri, Jun 8, 2018 at 9:44 AM, Yaron Koren  wrote:

> I suppose that one solution which hasn't been discussed yet is to change
> the wording of that file so that it says something more defensible, like
> "This extension is hosted on facilities governed by the Code of Conduct",
> or that kind of thing - that would at least remove the pragmatic objection
> that I (and some others in this thread) have raised.
>

Let's not lose sight of Yaron's proposal for compromise here.  Having
slightly different wording here seems like it would be a win-win for
everyone: those who want to be sure new contributors get a pointer to the
CoC would be satisfied, and Yaron would be satisfied that he is not
misrepresenting the scope of the CoC in his own repositories.

It seems that other multi-homed repositories might have similar wording
tweaks.  A repo which takes pull requests via github, for example, might
have a CoC mentioning the applicability of github's CoC or WMF's CoC,
depending on circumstances.

I think there are a number of fascinating topics here, and it's probably
certainly worth documenting the costs/benefits of WMF hosting (eg implicit
+2 access by maintainers, which could be a con, but the pro side is that
you get translatewiki integration, library upgrades, and other maintenance
work done "for free") along with pointers so that new repo owners can
easily find out exactly who has maintainer rights to their repo and a short
history describing why that decision was made.  That documentation would
also be a good place for best practices such as README and CoC (as well as
clarifying the circumstances where the CoC is expected to apply).


> It still leaves open the question of whether the file should be made
> mandatory, and if so, what the mechanism should be to determine that.
>

A good question. My personal opinion is (a) it should be mandatory, but the
contents not strictly proscribed -- in the same way that github
loosely-requires an open source license statement in its public repos (ie,
I think it's in the terms of service but not enforced by code), and (b) the
proper mechanism is probably techcomm/archcomm.  This is just my opinion --
I'm interested in hearing further discussion of these points --- but let's
not get distracted from considering (and hopefully accepting) Yaron's
compromise offer.
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] First step for MCR merged: Deprecate and gut the Revision class

[C. Scott Ananian]

It seems like this patch broke running `php tests/parser/parserTests.php`
directly?
See: https://phabricator.wikimedia.org/T184122
  --scott

On Sat, Dec 23, 2017 at 4:29 AM, Addshore <addshorew...@gmail.com> wrote:

> So, the patches that would need to be reverted can be found on wikitech
> https://wikitech.wikimedia.org/wiki/User:Addshore/MCR_Revert
>
> I have also created a patch with a switch wrapping the refactoring
> https://gerrit.wikimedia.org/r/#/c/399881/
>
> I'm going to continue testing this code on beta over the christmas period
> patching any holes that I find as I do.
>
> On 23 December 2017 at 00:14, Daniel Kinzler <daniel.kinz...@wikimedia.de>
> wrote:
>
>> Am 23.12.2017 um 00:03 schrieb C. Scott Ananian:
>> > I think a simple revert would be simplest.  Adding a feature flag adds
>> new
>> > possibilities of overlooked bugs, especially since this is "just" a
>> > refactoring and so *in theory* shouldn't be changing anything.
>> >
>> > Maybe we could just cherry-pick a revert onto the Jan 2 branch, rather
>> than
>> > revert on master and then un-revert after the branch.
>>
>> A revert is certainly an option, I tried to keep this as isolated as
>> possible.
>> Reverting just on the branch would allow us to keep testing on beta
>> without
>> disruption, and without having to go back and forth con core code,
>> causing merge
>> conflicts.
>>
>> But there is another option to consider: Only deploy to testwiki (and
>> friends)
>> on Jan 2, and not to production wikis. This would give us a week to look
>> at this
>> in a production-like environment, on top of the time on beta, before it
>> really
>> goes live.
>>
>> -- daniel
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>


-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] First step for MCR merged: Deprecate and gut the Revision class

[C. Scott Ananian]

I think a simple revert would be simplest.  Adding a feature flag adds new
possibilities of overlooked bugs, especially since this is "just" a
refactoring and so *in theory* shouldn't be changing anything.

Maybe we could just cherry-pick a revert onto the Jan 2 branch, rather than
revert on master and then un-revert after the branch.  It shouldn't be
interpreted as any comment on code quality or "readiness", just a
reflection of the fact that the first deploy after the holiday will
inevitable cause *some* breakage, and in our eggnog-induced stupor it would
be nice if we could just rule out this refactor as contributing to whatever
the breakage turns out to be.  (That's why a feature flag doesn't seem
helpful to me; it would still be lines of code to comb through.  Better
either a clean revert or just deploy the thing as is.)
 --scott

On Fri, Dec 22, 2017 at 2:01 PM, Addshore  wrote:

> So the plan going forward will be to create a feature flag for the MCR
> Revision gutting.
> I'll crack on with that this evening.
>
> If that turns out to be too messy then we can revert the MCR patches for
> the next wmf branch.
> I'm currently keeping track of this @
> https://wikitech.wikimedia.org/wiki/User:Addshore/MCR_Revert
>
> On 22 December 2017 at 18:39, Ramsey Isler  wrote:
>
> > Fantastic news! Great work handling this behemoth of a technical
> challenge.
> >
> > On Fri, Dec 22, 2017 at 2:26 AM, Daniel Kinzler <
> > daniel.kinz...@wikimedia.de> wrote:
> >
> >> Hello all!
> >>
> >> Addshore last night merged the patch[1] that is the first major step
> >> towards
> >> Multi-Content-Revisions[2]: it completely guts the Revision class and
> >> turns it
> >> into a thin proxy for the new RevisionStore service. The new code is now
> >> live
> >> on beta.
> >>
> >> This is our second attempt: The first one, on December 18th, thoroughly
> >> corrupted the beta database. It took us some time and a lot of help from
> >> Aaron
> >> and especially Roan to figure out what was happening. A detailed
> >> post-mortem by
> >> Roan can be found at [3].
> >>
> >> Anyway - this stage of MCR development introduces the new multi-revision
> >> capable
> >> interface for revision storage (and blob storage) [4]. It does not yet
> >> introduce
> >> the new database schema, that will be the next step [5][6]. While doing
> >> the
> >> refactoring, I tried to keep the structure of the existing code mostly
> >> intact,
> >> just moving functionality out of Revision into the new classes, most
> >> importantly
> >> RevisionRecord, RevisionStore, and BlobStore.
> >>
> >> Beware that with the next deployment (due January 2nd) the live sites
> >> will start
> >> using the new code. Please keep an eye out for any strangeness regarding
> >> revision handling. Adam greatly improved test coverage of the relevant
> >> code
> >> (thanks Adam!), but it's always possible that we missed some edge case,
> >> maybe
> >> something about archived revisions that were partially migrated from on
> >> old
> >> schema or something similarly fun.
> >>
> >> Exiting times!
> >>
> >> Cheers
> >> Daniel
> >>
> >>
> >> [1] https://gerrit.wikimedia.org/r/#/c/399174/
> >> [2] https://www.mediawiki.org/wiki/Requests_for_comment/Multi-
> >> Content_Revisions
> >> [3] https://phabricator.wikimedia.org/T183252#3853749
> >> [4] https://phabricator.wikimedia.org/T174025
> >> [5] https://phabricator.wikimedia.org/T174024
> >> [6] https://phabricator.wikimedia.org/T174030
> >>
> >>
> >> --
> >> Daniel Kinzler
> >> Principal Platform Engineer
> >>
> >> Wikimedia Deutschland
> >> Gesellschaft zur Förderung Freien Wissens e.V.
> >>
> >>
> >> ___
> >> Wikitech-l mailing list
> >> Wikitech-l@lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] First step for MCR merged: Deprecate and gut the Revision class

[C. Scott Ananian]

Having just read through T183252, I feel Roan deserves a big hand for his
"I take a walk and become Sherlock Holmes" detective work and "I'm just
like Indiana Jones, except with code not tombs and bugs not snakes" code
archaeology.

I love working with smart folks.
  --scott

On Fri, Dec 22, 2017 at 11:37 AM, Chad  wrote:

> Considering the code just landed last night and a good number of us are
> going to be gone for vacation--is rolling this out with the Jan 2nd deploy
> a little aggressive? I'd love to see this sit on beta (with eyes on it) for
> a little longer. Or a way to deploy to testwiki etc independent of major
> sites?
>
> The first deploy after a holiday break is already pretty exciting, and
> rolling this out feels like something that could use a dedicated window.
>
> (Otherwise, kudos to the MCR team for hitting this milestone)
>
> -Chad
>
> On Fri, Dec 22, 2017 at 2:27 AM Daniel Kinzler <
> daniel.kinz...@wikimedia.de>
> wrote:
>
> > Hello all!
> >
> > Addshore last night merged the patch[1] that is the first major step
> > towards
> > Multi-Content-Revisions[2]: it completely guts the Revision class and
> > turns it
> > into a thin proxy for the new RevisionStore service. The new code is now
> > live
> > on beta.
> >
> > This is our second attempt: The first one, on December 18th, thoroughly
> > corrupted the beta database. It took us some time and a lot of help from
> > Aaron
> > and especially Roan to figure out what was happening. A detailed
> > post-mortem by
> > Roan can be found at [3].
> >
> > Anyway - this stage of MCR development introduces the new multi-revision
> > capable
> > interface for revision storage (and blob storage) [4]. It does not yet
> > introduce
> > the new database schema, that will be the next step [5][6]. While doing
> the
> > refactoring, I tried to keep the structure of the existing code mostly
> > intact,
> > just moving functionality out of Revision into the new classes, most
> > importantly
> > RevisionRecord, RevisionStore, and BlobStore.
> >
> > Beware that with the next deployment (due January 2nd) the live sites
> will
> > start
> > using the new code. Please keep an eye out for any strangeness regarding
> > revision handling. Adam greatly improved test coverage of the relevant
> code
> > (thanks Adam!), but it's always possible that we missed some edge case,
> > maybe
> > something about archived revisions that were partially migrated from on
> old
> > schema or something similarly fun.
> >
> > Exiting times!
> >
> > Cheers
> > Daniel
> >
> >
> > [1] https://gerrit.wikimedia.org/r/#/c/399174/
> > [2]
> > https://www.mediawiki.org/wiki/Requests_for_comment/
> Multi-Content_Revisions
> > [3] https://phabricator.wikimedia.org/T183252#3853749
> > [4] https://phabricator.wikimedia.org/T174025
> > [5] https://phabricator.wikimedia.org/T174024
> > [6] https://phabricator.wikimedia.org/T174030
> >
> >
> > --
> > Daniel Kinzler
> > Principal Platform Engineer
> >
> > Wikimedia Deutschland
> > Gesellschaft zur Förderung Freien Wissens e.V.
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Person of interest to blame

[C. Scott Ananian]

git log includes/specials/SpecialWatchlist.php suggests:

T172757 / I27bdaa1401323fa3143f79a57dc5b9773e48fd1d (sep 27)
T176857 / I90c79ef9c4819a34060515b863277fd185828ed9 (sep 27)
T176348 / If6280ad6fbad65909e1d0b2a48344e24d485aca2 (sep 21)
T173533 / I4138fde22bdd8cc55c65846b91184c3ad3057244 (sep 19)
T176155 / Idc4e450c9fa03fc78a012fb5eefa60174bb1245a (sep 18)
T175965 / I14ba792f1cd435f89b2e09067b0a0e894a0a2557 (sep 14)
T168376 / I3c75f9f2f6287414bf330f116d959d078250392d (aug 23)
T174725 / I1cb402559edb242b40a77f509480560f0636264d (sep 1)
T172030 / I65c50d75b21b3e5d7b69fdecb79c032f5d1d8853 (aug 31)
-  / I1c50b67b0a4aaddb561e79fdacd9849ffe0ded8c (aug 31)
and a bunch more in late august related to "WLFilters".

As Sebastien suggests, the easiest way to identify the problem is to run a
local copy and use git-bisect to identify exactly where the problematic
change occurred.
 --scott


On Tue, Oct 3, 2017 at 2:37 PM, יגאל חיטרון  wrote:

> Hello. There was a patch, or a bug, or a bug fix in the last month that I'm
> trying to find. Something changed the Special:Watchlist view and broke a
> couple of gadgets. I wanted to find what was the gerrit patch, or at least
> some explanation, what exactly was done, but nobody knows (Thanks a lot to
> Quiddity which spent many time trying to help, you are awesome).
> The problem is different HTML structure of Special:Watchlist, at least when
> "Group changes by page in recent changes and watchlist" in preferences is
> on. I even can't describe the difference, because it's just "looks
> different". I read a roadmap for the last 50 days, and can't find anything.
> A couple of people I asked can't help. One of the symptoms is different css
> inside the revisions groups, bold vs unbold.
> Maybe somebody can help to find a gerrit to blame? Thank you,
> Igal (User:IKhitron)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Sep 19, 2017 9:45 PM, "Tim Starling"  wrote:

Facebook have been inconsistent with
HHVM, and have made it clear that they don't intend to cater to our needs.


I'm curious: is this conclusion based on your recent meeting with them, or
on past behavior?  Their recent announcement had a lot of "we know we
haven't been great, but we promise to change" stuff in it ("reinvest in
open source") and I'm curious to know if they enumerated concrete steps
they planned to take, or whether even in your most recent meeting with them
they failed to show actual interest.
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Tue, Sep 19, 2017 at 2:41 PM, C. Scott Ananian <canan...@wikimedia.org>
wrote:

> source".  You also mentioned PHP's long history of FLOSS without also
> mentioning their long history at sucking at security.
>

Whoops, I should have toned that down a bit before hitting send.  To be
clear, I'm mostly talking about the ~2007 time frame where there was a lot
of tension between the PHP core team and various folks who wanted to make
PHP more secure in different ways.  I don't actually know what the
present-day status is -- suhosin seems to be still around, but (for
instance) https://sektioneins.de/en/categories/php.html hasn't had any
particular complaints since 2015.

So to be super clear: I'm just pointing out that there used to be issues
here; sometimes the community's interests do not exactly align.  Consider
me in the devil's advocate role again: I'd be interested to hear an
insider's opinion (Stas?) on how security issues are handled these days and
what the future outlook is,
https://www.cvedetails.com/vulnerability-list/vendor_id-74/product_id-128/PHP-PHP.html
doesn't look as nice as
http://www.cvedetails.com/vulnerability-list/vendor_id-7758/product_id-35896/Facebook-Hhvm.html
but
maybe the latter is misleading; older vulnerabilities seem to be at
http://www.cvedetails.com/vulnerability-list/vendor_id-7758/product_id-30684/Facebook-Hiphop-Virtual-Machine.html
for instance.
  --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

Chad: thanks for writing up all of those in one place.  I'm not actually
trying to argue *for* Hack here.  My argument is just that we should take
the time to lay out the arguments carefully on both sides, since this is a
decision that is likely to have long-lasting effects.  Think of this like
the traditional [[devil's advocate]] process for canonizing a saint.  I'm
just challenging us to take a hard look at our reasoning and make sure it
is well-founded technically.

In my ideal world we'd take your excellent point-by-point argument and try
to rigorously quantify each, just to satisfy ourselves that we've done due
diligence.  You say "there's not much migration cost moving to PHP7" --
well, it would be nice to assign someone to go through the details in a
little more detail to double-check that.  "Various benchmarks dhow HHVM and
PHP7 performance to be roughly on par" -- firmer numbers would be nice, as
you say.  I think Tim was already going to do this?  "A bunch of major
projects have already dropped HHVM" -- again, can we take the devil's
advocate position and ask the HHVM team who *hasn't* dropped HHVM ;) and
ask for their reasons?  You also raise some points about packaging and
portability that I could probably counter by pointing out that HHVM has an
ARM and PowerPC JIT now (
http://hhvm.com/blog/2017/03/09/how-the-cyber-elephant-got-his-arm.html),
while Zend's JIT-in-progress only supports x86.  We could also poke at the
HHVM folks and ask about their packaging plans, since they (according to
their announcement) are going to "reinvest in open source".  You also
mentioned PHP's long history of FLOSS without also mentioning their long
history at sucking at security.  And we could try to quantify how dependent
PHP is on Zend contributions vs how dependent HHVM is on facebook
contributions.  We can get numbers on this, we don't have to assume things.

The HHVM announcement specifically mentioned that they will maintain
compatibility with composer and phpunit, so that seems to be a wash.

The references/destructors thing is actually really interesting as a
technical discussion.  The reason given for removing these is all about
performance, in particular the performance hit you get from ref-counting as
opposed to pure GC.  There's been a lot of really good GC work done in the
past two decades; see https://news.ycombinator.com/item?id=11759762
although I like to cite https://www.cs.princeton.edu/~appel/papers/45.ps as
foundational.  It may be a good opportunity to take a hard look at our
Hooks system and figure out if its design is future-proof.
  --scott

On Tue, Sep 19, 2017 at 2:04 PM, Chad <innocentkil...@gmail.com> wrote:

> On Tue, Sep 19, 2017 at 10:50 AM C. Scott Ananian <canan...@wikimedia.org>
> wrote:
>
> > Chad: the more you argue that it's not even worth considering, the more
> I'm
> > going to push back and ask for actual numbers and facts. ;)
> >
> >
> As I said in my prior message: it's been considered, and discarded rather
> quickly. It doesn't need much further introspection than that. A couple of
> major points:
>
> * Brian W's point is correct: there's not much migration cost at all moving
> to PHP7. Moving between PHP versions has always been pretty easy for us in
> MW--we write very conservative code as it is.
> * Various (there's lots from lots of people) benchmarks routinely show HHVM
> and PHP7--especially 7.1+--performance to be roughly on par, depending on
> workloads. We can get some firmer numbers here.
> ** A bunch of major projects have already dropped HHVM for PHP7+. Etsy,
> Symphony, Wordpress (they no longer test against it)
> * Swapping to HHVM/Hack would abandon many/most of our downstream users --
> the stats Stas pointed to way far back shows world install base as well
> under 0.5% of all PHP runtimes.
> * Speaking of downstream users: HHVM has always been a second-class citizen
> on non-Linux OSes. It's always been wonky on OSX (I would know, I was the
> first person to ever get it built), and I don't think anyone's ever got it
> working on Windows outside of Cygwin--please correct me if I'm wrong?
> * It greatly complicates setup and development work for both WMF and
> volunteers.
> ** Did I mention almost nobody has up to date packages for this?
> *  PHP has a long-documented history of working as a proper FLOSS project.
> Facebook's track record here has been less than stellar--it comes in fits
> and starts and there's a lot of "throwing code over the wall"
> ** We have a core PHP contributor (Stas) on this list. Other than
> occasional patches we've shipped upstream to HHVM, I doubt anyone would
> claim themselves a core HHVM contributor around here [maybe Tim early on,
> heh]
> * Choices HHVM is going to make upstream (references, destuctors) are a
> /huge/ issue for us.

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

Chad: the more you argue that it's not even worth considering, the more I'm
going to push back and ask for actual numbers and facts. ;)

Alex: I admit 8 contributors is not significant, but 24,192 commits vs
104,566 commits is. The conclusion I was suggesting was "same number of
contributors over a much shorter time frame".

Re: github mirror broken: note that http://php.net/build-setup.php mentions
*only* the github mirror.  And the github mirror has been broken for almost
two months now.  There's no mention of the "real" repo.  That merits an
exclamation mark, I think.
  --scott
​
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Tue, Sep 19, 2017 at 1:17 PM, Chad  wrote:

> be clear: we never chose HHVM for Hack. We don't use Hack. The one
> experiment I had at trying Hack never panned out. MediaWiki is in PHP, not
> Hack.
>

To be super clear: MediaWiki is in *PHP5*.  The choices are:

1) MediaWiki will always be in PHP5.
2) MediaWiki will eventually migrate to PHP7, or
3) MediaWiki will eventually migrate to Hack.

Is anyone arguing for #1?

So we've got two backwards-incompatible choices to make, eventually.
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Tue, Sep 19, 2017 at 11:34 AM, Bryan Davis <bd...@wikimedia.org> wrote:

> On Tue, Sep 19, 2017 at 9:21 AM, C. Scott Ananian
> <canan...@wikimedia.org> wrote:
> > There are other big users of HHVM -- do we know what other members of the
> > larger community are doing?  We've heard that Phabricator intends to
> follow
> > PHP 7.  Etsy also shifted to HHVM, do we know what their plans are?
>
> Etsy 'experimented' with HHVM [0] and then eventually switched to PHP7
> as their primary runtime. The blog posts about this are a little
> scattered, but Rasmus spoke about it [1] and Etsy started the phan
> project [2].
>

I got confirmation on twitter:
https://twitter.com/jazzdan/status/910162545805336576


> For what it's worth, my opinion is that PHP is an actual FLOSS
> software project with years of history and core contributions from
> Zend who make their living with PHP. HHVM is a well funded internal
> project from Facebook that has experimented with FLOSS but ultimately
> is controlled by the internal needs of Facebook. For me the choice
> here is obviously to back the community driven FLOSS project and help
> them continue to thrive.
>

Fair enough.  My point is just that we should stop and reflect that this is
a major inflection point.  Language choices are sticky, so this decision
will have significant long-term implications.  We should at least stop to
evaluate PHP7 vs Hack and determine which is a better fit for our codebase,
and do due diligence on both sides (count how many engineers, how many open
source contributors, commit rates, etc).  HHVM has been flirting with a
LLVM backend, and LLVM itself has quite a large and active community.  The
PHP community has had issues with proper handling of security patches in
the past.  I'm suggesting to proceed cautiously and have a proper
discussion of all the factors involved instead of over-simplifying this to
"community" vs "facebook".

For example, the top-line github stats are:
hhvm: 504 contributors (24,192 commits)
php-src: 496 contributors (104,566 commits)

HHVM seems to have a larger community of contributors despite a much
shorter active life.  But note that the PHP github mirror has been broken
since Jul 29 (!).  In the past 6 days I count 8 distinct contributors to
php-src, and 10 distinct contributors in the past *two days* to hhvm (one
of whom contributed an OCAML frontend(!)).  These are just hand-wavy
figures; ideally we should try to determine how many of the recent
contributors to each project are employed by Facebook and/or Zend.

I think there's room for a reasonable debate.
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Tue, Sep 19, 2017 at 11:33 AM, bawolff  wrote:

> I disagree. I don't think its useful or possible to try to forecast
> technical trends 15 years out. 15 years from now, it is just as likely
> that facebook will be as relevant as myspace is today, as it is that
> facebook will go full cyberpunk dystopia on us and rule the world. I
> don't think we can realistically predict anything 15 years out.
>
>
That's why I argued we should consider the code quality of both, and
determine which we'd feel most comfortable supporting on our own in the
case our crystal balls are wrong and we end up holding the bag for the
runtime.  (This is just one factor to consider; my argument is that we
should carefully enumerate the various factors before making an evaluation,
not just dismiss any of the options out of hand.)
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Mon, Sep 18, 2017 at 9:51 PM, Chad  wrote:

> I see zero reason for us to go through all the formalities, unless we want
> to really. I have yet to see anyone (on list, or on IRC anywhere at all
> today) where anyone suggested (2) was a good idea at all. It's a
> horrifically bad idea.
>

Technically, I did outline the arguments for (2), earlier on this thread.
It was a bit allegorical, though:
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2018/Writing_Tips/Examples#Example_of_a_.22medium_level.22_position_statement
This should be seriously considered, not just dismissed.

I agree the short timeline seems to push us toward reverting to Zend.  But
it is worth having a meaningful discussion about the long-term outlook.
Which VM is likely to be better supported in 15 years' time?  Which VM
would we rather adopt and maintain indefinitely ourselves, if needed --
since in a 15 yr timeframe it's entirely possible that (a) Facebook could
abandon Hack/HHVM, or (b) the PHP Zend team could implode.  Maintaining
control over our core runtime is important; I think we should at least
discuss long-term contingencies if either goes down. Obviously, our future
was most stable when we (briefly!) had a choice between two strong
runtimes... but that opportunity seems to be vanishing.  Practically
speaking, it's not really a choice between "lock-in" and "no lock in" -- we
have to choose to align our futures with either Zend Technologies Ltd or
Facebook.  One of these is *much* better funded than the other.  It is
likely that the project with the most funding will continue to have the
better performance.

There are other big users of HHVM -- do we know what other members of the
larger community are doing?  We've heard that Phabricator intends to follow
PHP 7.  Etsy also shifted to HHVM, do we know what their plans are?
  --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

Other tools we are using, such as Phabricator, will also be following HHVM
to Hack (presumably).  Facebook, as the largest (by engineering budget)
production user of PHP, will certainly have an outsized influence on the
direction of the platform and surrounding ecosystem.  If we follow path #3
we're probably also committing to supporting Zend development long-term as
the primary production user.
  --scott

On Mon, Sep 18, 2017 at 5:26 PM, Brian Wolff  wrote:

> On Monday, September 18, 2017, Max Semenik  wrote:
> > Today, the HHVM developers made an announcement[1] that they have plans
> of
> > ceasing to maintain 100% PHP7 compatibility and concentrating on Hack
> > instead.
> >
> > While this does not mean that we need to take an action immediately,
> > eventually we will have to decide something. As I see it, our options
> are:
> >
> > 1) Continue requiring that MediaWiki uses a common set of HHVM and Zend
> > PHP. This, however, is a dead end and will make things progressively
> harder
> > as the implementations will diverge and various Composer libraries we use
> > will start requiring some Zend-specific features.
> >
> > 2) Declare our loyalty to HHVM. This will result in most of our current
> > users being unable to upgrade, eventually producing what amounts to a
> > WMF-only product and lots of installations with outdated MediaWiki having
> > security holes. At least we will be able to convert to Hack eventually.
> > This is a very clean-cut case of vendor lock-in though, and if Facebook
> > decides to switch their code base to something shinier, we'll be deep in
> > trouble.
> >
> > 3) Revert WMF to Zend and forget about HHVM. This will result in
> > performance degradation, however it will not be that dramatic: when we
> > upgraded, we switched to HHVM from PHP 5.3 which was really outdated,
> while
> > 5.6 and 7 provided nice performance improvements.
> >
> > I personally think that 3) is the only viable option in the long run.
> What
> > do you think?
> >
> > 
> > [1] http://hhvm.com/blog/2017/09/18/the-future-of-hhvm.html
> >
> > --
> > Best regards,
> > Max Semenik ([[User:MaxSem]])
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> Well i agree that 3 seems likely the best long term option, we will
> probably be doing 1 in the short term. However I think it would be prudent
> to wait and see how things turn out in the short term before comitting to
> any path. The landscape can still shift quite a lot before the time comes
> when its impractical to continue doing 1.
>
> --
> bawolff
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[C. Scott Ananian]

On Mon, Sep 18, 2017 at 4:58 PM, Max Semenik  wrote:

> security holes. At least we will be able to convert to Hack eventually.
> This is a very clean-cut case of vendor lock-in though, and if Facebook
> decides to switch their code base to something shinier, we'll be deep in
> trouble.
>

I actually predicted this and wrote up my suggestions here:
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2018/Writing_Tips/Examples#Example_of_a_.22medium_level.22_position_statement

I will *not* be making this my position statement at the dev summit -- only
one per person, so many positions to take, this one didn't make the cut! --
but I would like to strongly encourage someone to submit a position
statement relating to this issue (could be any of max's three positions) to
the dev summit.  I think it is an important issue for our foundation and
community to discuss; certainly very relevant to the next 15 years of the
project.
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Can we drop revision hashes (rev_sha1)?

[C. Scott Ananian]

Alternatively, perhaps "hash" could be an optional part of an MCR chunk?
We could keep it for the wikitext, but drop the hash for the metadata, and
drop any support for a "combined" hash over wikitext + all-other-pieces.

...which begs the question about how reverts work in MCR.  Is it just the
wikitext which is reverted, or do categories and other metadata revert as
well?  And perhaps we can just mark these at revert time instead of trying
to reconstruct it after the fact?
 --scott

On Fri, Sep 15, 2017 at 4:13 PM, Stas Malyshev 
wrote:

> Hi!
>
> On 9/15/17 1:06 PM, Andrew Otto wrote:
> >> As a random idea - would it be possible to calculate the hashes
> > when data is transitioned from SQL to Hadoop storage?
> >
> > We take monthly snapshots of the entire history, so every month we’d
> > have to pull the content of every revision ever made :o
>
> Why? If you already seen that revision in previous snapshot, you'd
> already have its hash? Admittedly, I have no idea how the process works,
> so I am just talking out of general knowledge and may miss some things.
> Also of course you already have hashes from revs till this day and up to
> the day we decide to turn the hash off. Starting that day, it'd have to
> be generated, but I see no reason to generate one more than once?
> --
> Stas Malyshev
> smalys...@wikimedia.org
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] @author annotations in files in the mediawiki codebase

[C. Scott Ananian]

It's the internet, nobody ever really goes away. ;)

I will say, this convention makes more sense for a large project like
mediawiki core with many contributors; even more so if some of those
contributors are volunteers or short-term, since the @author tag can help
to track them down absent any long-term association with the WMF.  It is
less necessary for a project like parsoid, which is authored in the main by
five or fewer folks.

But I'd support an effort to add @author tags which deference via the wiki
module owners table.
  --scott

On Jun 13, 2017 12:59 PM, "Subramanya Sastry" <ssas...@wikimedia.org> wrote:

> On 06/13/2017 10:14 AM, C. Scott Ananian wrote:
>
> On Jun 13, 2017 6:24 AM, "Gergo Tisza" <gti...@wikimedia.org> wrote:
>>
>> On Tue, Jun 13, 2017 at 7:11 AM, Subramanya Sastry <ssas...@wikimedia.org
>> >
>> wrote:
>>
>> I find these annotations misleading and wonder why they exist and what
>>> purpose they serve.
>>>
>>
>> > It can sometimes tell you whom to ask for advice or reviews. (git log
>> would too but it's more effort.)
>>
>>
>> For the record (since one of my patches was specifically mentioned)
>> Gergo's
>> reason matches mine.  This is common (if informal) practice for a number
>> of
>> open source software projects---a way to easily indicate the original
>> author of the code, as a pointer to who to ask if you've got questions
>> about it.  T
>>
>
> I think the @author tag is at best a documentation hack for this scenario.
> What happens when people leave the project or there are more than one
> person who understands that code, or expertise shifts with changing
> codebase?
>
> It would be better to actually add a documentation line to point to a wiki
> page where this information can be found (and kept up to date).
>
> Subbu.
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] @author annotations in files in the mediawiki codebase

[C. Scott Ananian]

On Jun 13, 2017 6:24 AM, "Gergo Tisza"  wrote:

On Tue, Jun 13, 2017 at 7:11 AM, Subramanya Sastry 
wrote:

> I find these annotations misleading and wonder why they exist and what
> purpose they serve.


It can sometimes tell you whom to ask for advice or reviews. (git log would
too but it's more effort.)


For the record (since one of my patches was specifically mentioned) Gergo's
reason matches mine.  This is common (if informal) practice for a number of
open source software projects---a way to easily indicate the original
author of the code, as a pointer to who to ask if you've got questions
about it.  The informal practice is to add yourself to the list if you
undertake any major change or refactoring, some contribution which would
make you "at least as good a person to ask questions of" for some part of
the file.

It's admittedly imperfect.  We have a list of module owners on-wiki which
is "better" but much harder to find.  And as Gergo says, `git log` is more
authoritative, although anyone who has tried this sort of archeology knows
that it can be exhausting to scroll through the long history of trivial
fixes, language updates, code style tweaks, etc to find the few people who
made significant contributions.

In my experience a central CREDITS file is actually worse for code
archeology (although perhaps meets certain legal obligations better).
Because it is separate from the file itself, it is harder to find (and if
it's hard to find, we might as well use our on-wiki list), and less likely
to be kept up to date. I'm listed in the Linux kernel CREDITS file for
several projects which have subsequently been completely removed from the
source tree! If you keep @authors in the source file at least it stands a
better shot at being removed if/when the code is, or of surviving a
renaming or refactoring if still relevant. (The linux kernel has both
standalone CREDITS akin to our on-wiki list of module maintainers as well
as in-flight authorship info, and I still get occasional inquiries from
folks diving into the pty code for the first time.)

If we were to make a change, I'd suggest including an @authors tag in each
file explicitly naming the appropriate on-wiki module owners list, ie:
  @author https://meta/wiki/Module_owners#Parser

Or what have you.  Simply removing the @author tag seems like it is
removing a resource for new contributors with no replacement.
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Setting up multiple Parsoid servers behind load balancer

[C. Scott Ananian]

It may be a lock issue. IIRC mediawiki can invoke parsoid which can then
reinvoke the mediawiki api.  I remember there being some corner case with
locking which caused this recursive invocation to deadlock in some (not
anything like production) situations.  If you can get a trace from the
"slow" mediawiki perhaps you will find it waiting for a lock to time out.
  --scott

On Jun 9, 2017 4:29 PM, "James Montalvo"  wrote:

> Thanks to everyone for all the responses. I'm learning a lot.
>
>
> In the short term we need to figure out how to make this work without
> RESTBase, but I've been convinced by this email chain that in the long term
> we'll need to incorporate RESTBase into our setup.
>
>
> At this point I think I've determined that the problem we're having is not
> actually a Parsoid problem, but somehow related to MediaWiki Core (PHP)
> response times. Something about my multi-server setup is causing 25% of MW
> core response times to be 25x longer than normal. I didn't notice this in
> my dev setup, prior to testing Parsoid, probably because I just assumed my
> laptop was old and underpowered. In other words, normal page loads were
> slower but I just figured that having multiple VMs up on my laptop
> functioning as full app servers was the reason. Parsoid evidently has a
> default timeout short enough that when Parsoid makes MW core API requests I
> was getting failures, causing me to misinterpret it as a Parsoid issue.
>
>
> To ensure it was not my underpowered laptop I moved my testing to a machine
> with 12 CPUs and 64 GB RAM.
>
>
> Our configuration script that allows us to define our setup as follows:
>
>
> load balancers = list, of, IP, addresses, ...
>
> app servers = list, of, IP, addresses, ...
>
> memcached servers = list, of, IP, addresses, ...
>
> db master = a.single.ip.address
>
> db replicas = list, of, IP, addresses, ...
>
> parsoid servers = list, of, IP, addresses, ...
>
> elasticsearch servers = list, of, IP, addresses, ...
>
>
> I have not run it with that many servers yet, but it's theoretically
> possible. A single server does not need to fill a single role, so in
> testing thus far my configs look more like:
>
>
> load balancers = server.3.ip.addr
>
> app servers = server.1.ip.addr, server.2.ip.addr
>
> memcached servers = server.1.ip.addr, server.2.ip.addr
>
> db master = server.1.ip.addr
>
> db replicas = server.2.ip.addr
>
> parsoid servers = server.1.ip.addr, server.2.ip.addr
>
> elasticsearch servers = server.1.ip.addr, server.2.ip.addr
>
>
> In short: three servers, one exclusively a load balancer, two with
> everything installed albeit one acting as DB master and the other as DB
> replica.
>
>
> We're running this setup in production with all servers configured as
> "localhost", e.g. everything installed on one server.
>
>
> I'm pretty sure I've narrowed down the 25x-longer-response-times to being a
> multiple app-server problem because I can take the dev config above
> (server.1.ip.addr, server.2.ip.addr, server.3.ip.addr) and comment out
> various servers and re-run deploy. This allows me to quickly switch from a
> single app server to two, two DBs to one, etc. I see the issue with
> multiple app servers. I don't see it with a single app server, regardless
> of whether the other services have 1 or 2 servers.
>
>
> My LocalSettings.php files are are at [1] and [2] for dual app servers.
> These reference Extensions.php which _shouldn't_ have any impact but can be
> found at [3]. The files are written by Ansible and I'm kind of bad at
> getting the indenting correct...so, sorry about that if it looks funny. All
> of this is created by our project called meza [4]. We weren't really
> planning on announcing meza yet, but basically its purpose is to simplify
> MediaWiki install with all the bells and whistles for "enterprise"
> (whatever that means :) ) use cases. We've been running it on a single
> server for about a year, but need to migrate to a high availability setup
> to support 24/7 mission critical operations.
>
>
> Any ideas what may cause two load-balanced app servers to respond slowly
> 25% of the time?
>
>
> Thanks!
>
> --James
>
>
> [1]
> https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431
> 08#file-localsettings-app1-php
>
> [2]
> https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431
> 08#file-localsettings-app2-php
>
> [3]
> https://gist.github.com/jamesmontalvo3/5adf207623454c9eff98e93152b431
> 08#file-extensions-php
>
> [4] https://github.com/enterprisemediawiki/meza
>
> On Fri, Jun 9, 2017 at 12:57 PM, Subramanya Sastry 
> wrote:
>
> > On 06/09/2017 09:57 AM, Gabriel Wicke wrote:
> >
> > On Fri, Jun 9, 2017 at 12:56 AM, Alexandros Kosiaris <
> >> akosia...@wikimedia.org> wrote:
> >>
> >>> I also don't think you need RESTBase as long as you are willing to
> >>> wait for parsoid to finish parsing and returning the result.
> >>>
> >> Apart from performance, there is also 

Re: [Wikitech-l] Setting up multiple Parsoid servers behind load balancer

[C. Scott Ananian]

https://www.mediawiki.org/wiki/Parsoid/Setup/RESTBase

Cassandra is optional, for a small deployment the sqlite backend is
probably sufficient.  Cassandra is the "distributed DB" part, so if you
used cassandra you could set up multiple restbase clients.

RESTBase is for performance, not availability.  It moved the parse time to
"after the page is saved" instead of "when the user hits edit", which makes
the editing interface feel much more snappy.

None of mediawiki, Parsoid, or RESTBase store any state other than in their
DBs (MySQL for mediawiki or Cassandra/sqlite for RESTBase).  We use LVS for
load balancing (https://wikitech.wikimedia.org/wiki/LVS).  All the
mediawikis point at LVS, which chooses an appropriate RESTBase.  All the
RESTBase point at LVS, which chooses an appropriate Parsoid.  All the
Parsoids point to LVS, which will give them an arbitary mediawiki for
fetching wikitext, etc.
  --scott

On Thu, Jun 8, 2017 at 10:10 AM, James Montalvo <jamesmontal...@gmail.com>
wrote:

> I've read through the documentation I think you're talking about. It's kind
> of hard to determine where to start since the docs are spread out between
> multiple VE, Parsoid and RESTBase pages. Installing RESTBase is, as you
> say, straightforward (git clone, npm install, basically). Configuring is
> not clear to me, and without clear docs it's the kind of thing that takes
> hours of trial and error. Also some parts mention I need Cassandra but it's
> not clear if that's a hard requirement.
>
> If I want a highly available setup with multiple app servers and multiple
> Parsoid servers, would I install RESTBase alongside each Parsoid? How does
> communication between the multiple app and RB/Parsoid servers get
> configured? I feel like I'll be back in the same load balancing situation.
>
> --James
>
> On Jun 8, 2017 7:43 AM, "C. Scott Ananian" <canan...@wikimedia.org> wrote:
>
> RESTBase actually adds a lot of immediate performance, since it lets VE
> load the editable representation directly from cache, instead of requiring
> the editor to wait for Parsoid to parse the page before it can be edited.
> I documented the RESTBase install; it shouldn't actually be any more
> difficult than Parsoid.  They both use the same service runner framework
> now.
>
> At any rate: in your configurations you have URL and HTTPProxy set to the
> exact same string.  This is almost certainly not right.  I believe if you
> just omit the proxy lines entirely from the configuration you'll find
> things work as you expect.
>  --scott
>
> On Wed, Jun 7, 2017 at 11:30 PM, James Montalvo <jamesmontal...@gmail.com>
> wrote:
>
> > Setting up RESTBase is very involved. I'd really prefer not to add that
> > complexity at this time. Also I'm not sure at my scale RESTBase would
> > provide much performance benefit (though I don't know much about it so
> > that's just a hunch). The parsoid and VE configs have fields for proxy
> (as
> > shown in my snippets), so it seems like running them this way is
> intended.
> > Am I wrong?
> >
> > Thanks,
> > James
> >
> > On Jun 7, 2017 8:12 PM, "C. Scott Ananian" <canan...@wikimedia.org>
> wrote:
> >
> > > I think in general the first thing you should do for performance is set
> > up
> > > restbase in front of parsoid? Caching the parsoid results will be
> faster
> > > than running multiple parsoids in parallel.  That would also match the
> > wmf
> > > configuration more closely, which would probably help us help you.  I
> > wrote
> > > up instructions for configuring restbase on the VE and Parsoid wiki
> > pages.
> > > As it turns out I updated these today to use VRS configuration. Let me
> > know
> > > if you run into trouble, perhaps some further minor updates are
> > necessary.
> > >   --scott
> > >
> > > On Jun 7, 2017 6:26 PM, "James Montalvo" <jamesmontal...@gmail.com>
> > wrote:
> > >
> > > > I'm trying to setup two Parsoid servers to play nicely with two
> > MediaWiki
> > > > application servers and am having some issues. I have no problem
> > getting
> > > > things working with Parsoid on a single app server, or multiple
> Parsoid
> > > > servers being used by a single app server, but ran into issues when I
> > > > increased to multiple app servers. To try to get this working I
> > starting
> > > > making the app and Parsoid servers communicate through my load
> > balancer.
> > > So
> > > > an overview of my config is:
> > > >
> > > > Load balancer 

Re: [Wikitech-l] Setting up multiple Parsoid servers behind load balancer

[C. Scott Ananian]

RESTBase actually adds a lot of immediate performance, since it lets VE
load the editable representation directly from cache, instead of requiring
the editor to wait for Parsoid to parse the page before it can be edited.
I documented the RESTBase install; it shouldn't actually be any more
difficult than Parsoid.  They both use the same service runner framework
now.

At any rate: in your configurations you have URL and HTTPProxy set to the
exact same string.  This is almost certainly not right.  I believe if you
just omit the proxy lines entirely from the configuration you'll find
things work as you expect.
 --scott

On Wed, Jun 7, 2017 at 11:30 PM, James Montalvo <jamesmontal...@gmail.com>
wrote:

> Setting up RESTBase is very involved. I'd really prefer not to add that
> complexity at this time. Also I'm not sure at my scale RESTBase would
> provide much performance benefit (though I don't know much about it so
> that's just a hunch). The parsoid and VE configs have fields for proxy (as
> shown in my snippets), so it seems like running them this way is intended.
> Am I wrong?
>
> Thanks,
> James
>
> On Jun 7, 2017 8:12 PM, "C. Scott Ananian" <canan...@wikimedia.org> wrote:
>
> > I think in general the first thing you should do for performance is set
> up
> > restbase in front of parsoid? Caching the parsoid results will be faster
> > than running multiple parsoids in parallel.  That would also match the
> wmf
> > configuration more closely, which would probably help us help you.  I
> wrote
> > up instructions for configuring restbase on the VE and Parsoid wiki
> pages.
> > As it turns out I updated these today to use VRS configuration. Let me
> know
> > if you run into trouble, perhaps some further minor updates are
> necessary.
> >   --scott
> >
> > On Jun 7, 2017 6:26 PM, "James Montalvo" <jamesmontal...@gmail.com>
> wrote:
> >
> > > I'm trying to setup two Parsoid servers to play nicely with two
> MediaWiki
> > > application servers and am having some issues. I have no problem
> getting
> > > things working with Parsoid on a single app server, or multiple Parsoid
> > > servers being used by a single app server, but ran into issues when I
> > > increased to multiple app servers. To try to get this working I
> starting
> > > making the app and Parsoid servers communicate through my load
> balancer.
> > So
> > > an overview of my config is:
> > >
> > > Load balancer = 192.168.56.63
> > >
> > > App1 = 192.168.56.80
> > > App2 = 192.168.56.60
> > >
> > > Parsoid1 = 192.168.56.80
> > > Parsoid2 = 192.168.56.60
> > >
> > > Note, App1 and Parsoid1 are the same server, and App2 and Parsoid2 are
> > the
> > > same server. I can only spin up so many VMs on my laptop.
> > >
> > > The load balancer (HAProxy) is configured as follows:
> > > * 80 forwards to 443
> > > * 443 forwards to App1 and App2 port 8080
> > > * 8081 forwards to App1 and App2 port 8080 (this will be a private
> > network
> > > connection later)
> > > * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be
> private)
> > >
> > > On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the
> > > appropriate response from Parsoid. I can run `curl 192.168.56.63:8081`
> > and
> > > get the appropriate response from MediaWiki. The same is true for both
> on
> > > App2/Parsoid2. So the servers can get the info they need from the
> > services.
> > >
> > > Currently I'm getting a the error "Error loading data from server: 500:
> > > docserver-http: HTTP 500. Would you like to retry?" when attempting to
> > use
> > > Visual Editor. I've tried various different settings and have not
> always
> > > gotten that specific error, but am getting it with the settings I
> > currently
> > > have in localsettings.js and LocalSettings.php (shown below in this
> > email).
> > > Removing the proxy config lines from these settings gave slightly
> better
> > > results. I did not get the 500 error, but instead it sometimes after a
> > very
> > > long time it would work. It also may have been throwing errors in the
> > > parsoid log (with debug on). I have those logs saved if they help. I'm
> > > hoping someone can just point out some misconfiguration, though.
> > >
> > > Here are snippets of my config files:
> > >
> > > On App1/Parsoid1, relevant localsettings.js:
> > >
> > > parsoidConfig.setMwApi

Re: [Wikitech-l] Setting up multiple Parsoid servers behind load balancer

[C. Scott Ananian]

I think in general the first thing you should do for performance is set up
restbase in front of parsoid? Caching the parsoid results will be faster
than running multiple parsoids in parallel.  That would also match the wmf
configuration more closely, which would probably help us help you.  I wrote
up instructions for configuring restbase on the VE and Parsoid wiki pages.
As it turns out I updated these today to use VRS configuration. Let me know
if you run into trouble, perhaps some further minor updates are necessary.
  --scott

On Jun 7, 2017 6:26 PM, "James Montalvo"  wrote:

> I'm trying to setup two Parsoid servers to play nicely with two MediaWiki
> application servers and am having some issues. I have no problem getting
> things working with Parsoid on a single app server, or multiple Parsoid
> servers being used by a single app server, but ran into issues when I
> increased to multiple app servers. To try to get this working I starting
> making the app and Parsoid servers communicate through my load balancer. So
> an overview of my config is:
>
> Load balancer = 192.168.56.63
>
> App1 = 192.168.56.80
> App2 = 192.168.56.60
>
> Parsoid1 = 192.168.56.80
> Parsoid2 = 192.168.56.60
>
> Note, App1 and Parsoid1 are the same server, and App2 and Parsoid2 are the
> same server. I can only spin up so many VMs on my laptop.
>
> The load balancer (HAProxy) is configured as follows:
> * 80 forwards to 443
> * 443 forwards to App1 and App2 port 8080
> * 8081 forwards to App1 and App2 port 8080 (this will be a private network
> connection later)
> * 8001 forwards to Parsoid1 and Parsoid2 port 8000 (also will be private)
>
> On App1/Parsoid1 I can run `curl 192.168.56.63:8001` and get the
> appropriate response from Parsoid. I can run `curl 192.168.56.63:8081` and
> get the appropriate response from MediaWiki. The same is true for both on
> App2/Parsoid2. So the servers can get the info they need from the services.
>
> Currently I'm getting a the error "Error loading data from server: 500:
> docserver-http: HTTP 500. Would you like to retry?" when attempting to use
> Visual Editor. I've tried various different settings and have not always
> gotten that specific error, but am getting it with the settings I currently
> have in localsettings.js and LocalSettings.php (shown below in this email).
> Removing the proxy config lines from these settings gave slightly better
> results. I did not get the 500 error, but instead it sometimes after a very
> long time it would work. It also may have been throwing errors in the
> parsoid log (with debug on). I have those logs saved if they help. I'm
> hoping someone can just point out some misconfiguration, though.
>
> Here are snippets of my config files:
>
> On App1/Parsoid1, relevant localsettings.js:
>
> parsoidConfig.setMwApi({
>
> uri: 'http://192.168.56.80:8081/demo/api.php',
> proxy: { uri: 'http://192.168.56.80:8081/' },
> domain: 'demo',
> prefix: 'demo'
> } );
>
> parsoidConfig.serverInterface = '192.168.56.80';
>
>
> On App2/Parsoid2, relevant localsettings.js:
>
> parsoidConfig.setMwApi({
>
> uri: 'http://192.168.56.80:8081/demo/api.php',
> proxy: { uri: 'http://192.168.56.80:8081/' },
>
> domain: 'demo',
> prefix: 'demo'
>
> } );
>
> parsoidConfig.serverInterface = '192.168.56.60';
>
>
> On App1/Parsoid1, relevant LocalSettings.php:
>
> $wgVirtualRestConfig['modules']['parsoid'] = array(
> 'url' => '192.168.56.80:8001',
>
> 'HTTPProxy' => 'http://192.168.56.80:8001',
>
> 'domain' => $wikiId,
> 'prefix' => $wikiId
> );
>
>
> On App2/Parsoid2, relevant LocalSettings.php:
>
> $wgVirtualRestConfig['modules']['parsoid'] = array(
> 'url' => '192.168.56.80:8001',
>
> 'HTTPProxy' => 'http://192.168.56.80:8001',
>
> 'domain' => $wikiId,
> 'prefix' => $wikiId
> );
>
>
> Thanks!
>
> --James
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Chinese conversion, search engines, and auto-detection

[C. Scott Ananian]

David or Liangent might be able to answer your questions.
  --scott

On May 17, 2017 12:01 PM, "Amir E. Aharoni" 
wrote:

> Hi,
>
> I wondered about some things around the Chinese variant conversion:
>
> * When a person uses a search engine, do the links in the results point
> directly to one of the variants? That is, does it point to
> https://zh.wikipedia.org/zh-cn/Article_name , etc., or simply to
> zh.wikipedia.org/wiki/Article_name ? I guess that among Chinese-speaking
> people Google is not necessarily as ubiquitous as elsewhere, so there is
> probably a separate answer for each search engine.
>
> * If for any search engine the answer above is "yes", does anybody have an
> idea about how does that search engine guess the preferred variant? Usage
> of simplified / traditional characters in the search query? Geolocation?
> Preferred language settings in the browser ("Accept-Language")? Preferences
> in the search engine itself? A combination of all of the above? Something
> else?
>
> * Does any of the search engine show direct links to country-based variants
> - zh-cn, zh-hk, zh-tw, zh-sg, zh-mo? Or to the more generic zh-hans and
> zh-hant?
>
> * For users who didn't log in, is the variant selection remembered in a
> cookie or in localStorage?
>
> I cannot easily test any of these things myself, because I don't speak
> Chinese, I'm not familiar with Chinese search engines, and I don't live in
> a Chinese-speaking country (and geolocation matters). But since I care
> about language, I'm very curious about this.
>
> Thanks!
>
> --
> Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
> http://aharoni.wordpress.com
> ‪“We're living in pieces,
> I want to live in peace.” – T. Moore‬
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Editing JSON in MediaWiki

[C. Scott Ananian]

...and inspired, I just submitted
https://wikimania2017.wikimedia.org/wiki/Submissions/Writing_Visual_Editor_and_Parsoid_extensions
  --scott

On Mon, Apr 10, 2017 at 3:15 PM, C. Scott Ananian <canan...@wikimedia.org>
wrote:

> More related functionality: Parsoid supports JSON as a contenttype, and
> emits a special type of table (the same as the HTML table generated by the
> HTML handler for JSON in mediawiki).  You can edit this in VE, although w/o
> any special support, and Parsoid will serialize it back to JSON.
>
> This could be turned into a very pleasant type-aware editor for JSON in VE
> pretty easily.
>  --scott
>
> On Sun, Apr 9, 2017 at 2:23 AM, Denny Vrandečić <vrande...@gmail.com>
> wrote:
>
>> Here's my requirement:
>> - a wiki page is one JSON document
>> - when editing, the user edits the JSON directly
>> - when viewing, I have a viewer that turns the JSON into wikitext, and
>> that
>> wikitext gets rendered as wikitext and turned into HTML by MediaWiki
>>
>> I have several options, including:
>>
>> 1) hook for a tag like , and write an extension that parses the
>> content between the tags and turns it into wikitext (not ideal, as I don't
>> use any of the existing support for JSON stuff, and also I could have
>> several such tags per page, which does not fit with my requirements)
>>
>> 2) I found the JsonConfig extension by yurik. This allows me to do almost
>> all of the things above - but it returns HTML directly, not wikitext. It
>> doesn't seem trivial to be able to return wikitext instead of HTML, but
>> hopefully I am wrong? Also, this ties in nicely with the Code Editor.
>>
>> 3) there is actually a JsonContentHandler in core. But looking through it
>> it seems that this suffers from the same limitations - I can return HTML,
>> but not wikitext.
>>
>> 3 seems to have the advantage to be more actively worked on that 2 (which
>> is not based on 3, probably because it is older than 3). So future goodies
>> like a Json Schema validator will probably go to 3, but not to 2, so I
>> should probably go to 3.
>>
>> Writing this down, one solution could be to create the wikitext, and then
>> call the wikitext parser manually and have it create HTML?
>>
>> I have already developed the extension in 1, and then fully rewritten it
>> in
>> 2.  Before I go and rewrite it again in 3, I wanted to ask whether I am
>> doing it right, or if should do it completely differently, and also if
>> there are examples of stuff developed in 3, i.e. of extensions or features
>> using the JsonContent class.
>>
>> Example:
>> I have a JSON document
>> { "username": "Denny" }
>> which gets turned into wikitext
>> ''Hello, [[User:Denny|Denny]]!''
>> which then gets turned into the right HTML and displayed to the user, e.g.
>> Hello, Denny!
>>
>> Cheers,
>> Denny
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
>
> --
> (http://cscott.net)
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Editing JSON with VisualEditor (was: Editing JSON in MediaWiki)

[C. Scott Ananian]

I was thinking of the pre-WIkimania hackathon.

There are a *lot* of extensions installed on WMF wikis.  Ideally we'd start
getting extension maintainers to take over the Parsoid/VE support for their
extensions.
 --scott

On Mon, Apr 10, 2017 at 3:37 PM, James Hare <jamesmh...@gmail.com> wrote:

> This upcoming hackathon might be too soon, considering we have more urgent
> priorities that need to be addressed first (like having it not be broken on
> the Beta Cluster). Beyond that, it’s something that can be considered, but
> how many people are actually interested in this?
>
>
> On April 10, 2017 at 12:26:26 PM, C. Scott Ananian (canan...@wikimedia.org)
> wrote:
>
> I *believe* so.  Both Parsoid and Visual Editor have extension
> interfaces.  However, they haven't really been used by folks outside WMF
> yet.  I'm eager to have that change.
>
> We're thinking about proposing a "Workshop" session at wikimania for "How
> to write extensions for Parsoid and Visual Editor".  But the area's only
> half-baked, it might be more appropriate to do it as a hackathon session
> instead.  Thoughts?
>   --scott
>
> On Mon, Apr 10, 2017 at 3:20 PM, James Hare <jamesmh...@gmail.com> wrote:
>
>> On April 10, 2017 at 12:15:34 PM, C. Scott Ananian (
>> canan...@wikimedia.org) wrote:
>>
>> More related functionality: Parsoid supports JSON as a contenttype, and
>> emits a special type of table (the same as the HTML table generated by the
>> HTML handler for JSON in mediawiki). You can edit this in VE, although w/o
>> any special support, and Parsoid will serialize it back to JSON.
>>
>> This could be turned into a very pleasant type-aware editor for JSON in VE
>> pretty easily.
>> --scott
>>
>>
>> This is actually very interesting.
>>
>> So, take a CollaborationHubContent page: https://wpx.wmflabs.org/
>> w/index.php/WPX:WikiProject_Dogs
>>
>> At some point we would like to create a VisualEditor-type editing
>> interface. Note that behind the scenes, that page is JSON. So in this
>> “VisualEditor” you would edit fields and such and it would appear you are
>> editing the page as though it were a regular document, but really you are
>> just changing values in a JSON blob.
>>
>> Since Parsoid supports JSON as a content type, would it thus be trivial
>> to create an extension for VisualEditor to support editing this content
>> type and others like it?
>>
>>
>> Thanks,
>> James Hare
>>
>
>
>
> --
> (http://cscott.net)
>
>


-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Editing JSON with VisualEditor (was: Editing JSON in MediaWiki)

[C. Scott Ananian]

I *believe* so.  Both Parsoid and Visual Editor have extension interfaces.
However, they haven't really been used by folks outside WMF yet.  I'm eager
to have that change.

We're thinking about proposing a "Workshop" session at wikimania for "How
to write extensions for Parsoid and Visual Editor".  But the area's only
half-baked, it might be more appropriate to do it as a hackathon session
instead.  Thoughts?
  --scott

On Mon, Apr 10, 2017 at 3:20 PM, James Hare <jamesmh...@gmail.com> wrote:

> On April 10, 2017 at 12:15:34 PM, C. Scott Ananian (canan...@wikimedia.org)
> wrote:
>
> More related functionality: Parsoid supports JSON as a contenttype, and
> emits a special type of table (the same as the HTML table generated by the
> HTML handler for JSON in mediawiki). You can edit this in VE, although w/o
> any special support, and Parsoid will serialize it back to JSON.
>
> This could be turned into a very pleasant type-aware editor for JSON in VE
> pretty easily.
> --scott
>
>
> This is actually very interesting.
>
> So, take a CollaborationHubContent page: https://wpx.wmflabs.org/
> w/index.php/WPX:WikiProject_Dogs
>
> At some point we would like to create a VisualEditor-type editing
> interface. Note that behind the scenes, that page is JSON. So in this
> “VisualEditor” you would edit fields and such and it would appear you are
> editing the page as though it were a regular document, but really you are
> just changing values in a JSON blob.
>
> Since Parsoid supports JSON as a content type, would it thus be trivial to
> create an extension for VisualEditor to support editing this content type
> and others like it?
>
>
> Thanks,
> James Hare
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Editing JSON in MediaWiki

[C. Scott Ananian]

More related functionality: Parsoid supports JSON as a contenttype, and
emits a special type of table (the same as the HTML table generated by the
HTML handler for JSON in mediawiki).  You can edit this in VE, although w/o
any special support, and Parsoid will serialize it back to JSON.

This could be turned into a very pleasant type-aware editor for JSON in VE
pretty easily.
 --scott

On Sun, Apr 9, 2017 at 2:23 AM, Denny Vrandečić  wrote:

> Here's my requirement:
> - a wiki page is one JSON document
> - when editing, the user edits the JSON directly
> - when viewing, I have a viewer that turns the JSON into wikitext, and that
> wikitext gets rendered as wikitext and turned into HTML by MediaWiki
>
> I have several options, including:
>
> 1) hook for a tag like , and write an extension that parses the
> content between the tags and turns it into wikitext (not ideal, as I don't
> use any of the existing support for JSON stuff, and also I could have
> several such tags per page, which does not fit with my requirements)
>
> 2) I found the JsonConfig extension by yurik. This allows me to do almost
> all of the things above - but it returns HTML directly, not wikitext. It
> doesn't seem trivial to be able to return wikitext instead of HTML, but
> hopefully I am wrong? Also, this ties in nicely with the Code Editor.
>
> 3) there is actually a JsonContentHandler in core. But looking through it
> it seems that this suffers from the same limitations - I can return HTML,
> but not wikitext.
>
> 3 seems to have the advantage to be more actively worked on that 2 (which
> is not based on 3, probably because it is older than 3). So future goodies
> like a Json Schema validator will probably go to 3, but not to 2, so I
> should probably go to 3.
>
> Writing this down, one solution could be to create the wikitext, and then
> call the wikitext parser manually and have it create HTML?
>
> I have already developed the extension in 1, and then fully rewritten it in
> 2.  Before I go and rewrite it again in 3, I wanted to ask whether I am
> doing it right, or if should do it completely differently, and also if
> there are examples of stuff developed in 3, i.e. of extensions or features
> using the JsonContent class.
>
> Example:
> I have a JSON document
> { "username": "Denny" }
> which gets turned into wikitext
> ''Hello, [[User:Denny|Denny]]!''
> which then gets turned into the right HTML and displayed to the user, e.g.
> Hello, Denny!
>
> Cheers,
> Denny
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Mediawiki-i18n] Offering internationalized programming facilities within WM enviroment

[C. Scott Ananian]

I think it is easier to create a new localizable programming environment
from scratch, than to try to localize existing tools and APIs.

For example, block-based programming languages (like Scratch and eToys)
tend to be fairly easy to translate -- there are some issues regarding
fixed size labels, the meaning of concatenating labels in various
langauges, etc, but these programs proved surmountable.  We used these
extensively at One Laptop per Child.

At OLPC I worked on a block-based JavaScript-subset, which allowed complete
translation between "text based" and "block based" views of the code:
http://turtlescript.github.cscott.net/
You could localize the labels in the block-based version and still "compile
down" to the legacy/English APIs.

As some folks know I've been a persistent advocate for JavaScript in
Scribunto, and I've contributed to v8js to this end.

Another option is to move away from textual scripting languages on wiki
entirely.  For example, https://phabricator.wikimedia.org/T114454 proposes
using Visual Editor to perform more of the "template" tasks, with a
stronger separation of code, "layout", and data.  Template edits which
affect the visual presentation or the data ("arguments") but not the
underlying code should be possible in Visual Editor in a fully localized
manner.  The textual "code" portion would be de-emphasized for most tasks.
For tasks which do require "code" to be written, you'd use one of the
techniques described above.
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Implementing Katherine's Vision: "Discussing Discussions"

[C. Scott Ananian]

A few weeks ago our Executive Director gave a talk on "Privacy and
Harassment on the Internet" at MozFest 2016 in London.  I encourage you to
read the transcript:

https://en.wikisource.org/wiki/Privacy_and_Harassment_on_the_Internet


Katherine argued that the Wikimedia project can take a lead role in
creating a culture of respect and inclusion online.  I whole-heartedly
agree, and I hope you all do too.  She concluded with:

"We have a lot of work to do. I know that. We know that. As Molly’s story
> illustrates, we are not there yet."


I'd like to open a broader discussion on how we get "there": how to
build/maintain places where we can get work done and control abuse and
vandalism while still remaining wide open to the universe of differing
viewpoints present in our projects.  We can't afford to create filter
bubbles, but we must be able to provide users safe(r) spaces to work.

By habit I would propose that this be a technical discussion, on specific
tools or features that our platform is currently missing to facilitate
healthy discussions.  But the "filter bubble" is a social problem, not a
technical one.  Our project isn't just a collection of code; it's a
community, a set of norms and habits, and a reflection of the social
process of collaboration.  A graph algorithm might be able to identify a
filter bubble and good UX can make countervailing opinions no more than a
click away, but it takes human will to seek out uncomfortable truth.

So although my endgame is specific engineering tasks, we need to start with
a broader conversation about our work as social creatures.  How do we work
in the projects, how do we communicate among ourselves, and how do we
balance openness and the pursuit of truth with the fight against abuse,
harassment, and bias.

Let's discuss discussions!

Here are some jumping off points; feel free to contribute your own:

We currently use a mixture of Talk pages, Echo, mailing lists, IRC,
Phabricator, OTRS, Slack, Conpherence, and Google Doc on our projects, with
different logging, publication, privacy/identity, and other
characteristics.  I tried to start cataloging them here:

https://lists.wikimedia.org/pipermail/wikimedia-l/2016-November/085542.html


Because of this diversity, we lack a unified code of conduct or mechanism
to report/combat harassment and vandalism.

Matt Flaschen replied in the above thread with an update on the Code of
Conduct for technical spaces:

https://lists.wikimedia.org/pipermail/wikimedia-l/2016-November/085542.html

...which should definitely help!  The creation of a centralized reporting
mechanism, in particular, would be most welcome.

I created a proposal for the Wikimedia Developer Summit in January
discussing "safe spaces" on our projects:

https://phabricator.wikimedia.org/T149665

Subscribe/comment/click "award token" to support its inclusion in the dev
summit or to start a conversation there.

I have another, broader, proposal as well, on the "future of chat" on our
projects:

https://phabricator.wikimedia.org/T149661

Subscribe/comment/click "award token" there if that angle piques your
interest.

It seems that "groups of users" arise repeatedly as an architectural
meta-concept, whether it's a group of collaborators you want to invite to
an editing session, a group of users you want to block or ban, a group of
users who belong to a particular wikiproject, or who watch a certain page.
We don't really have a first-class representation of that concept in our
code right now.  In previous conversations I've heard that people "don't
want  to turn into another facebook" and so have pushed
back strongly on the idea of "friend lists" (one type of group of users) --
but inverting the concept to allow WikiProjects to maintain a list of
"members of the wikiproject" is more palatable, more focused on the editing
task.  From a computer science perspective "friend list" and "member of a
wikiproject" might seem identical--they are both lists of users--but from a
social perspective the connotations and focus are significantly different.
But who administers that list of users?

Perhaps we can build a system which avoids grappling with user groups
entirely.  It was suggested that we might use an ORES-like system to
automatically suggest collaborators on an editing project based on some
criteria (like editing history), rather than force you or the WikiProject
to maintain an explicit list.  Perhaps you can implement block lists to
combat harassment based entirely on keywords, not users.  Do we trust the
machine to be more fair and less abusive than us mere mortals? Additional
ideas welcome!   (I don't have a dedicated phab task for this, but
https://phabricator.wikimedia.org/T149665 might be appropriate if you want
to contribute off-list.)


Hopefully this has been enough to prime the pump.

Let's discuss discussions.

Let's live up to the hope placed in us by the Washington Post:

Re: [Wikitech-l] Update on WMF account compromises

[C. Scott Ananian]

On Wed, Nov 16, 2016 at 1:19 PM, Pine W  wrote:

> (1) If you find it difficult to remember strong passwords then consider
> using a password manager .
>

For Unix/GPG types, I can recommend http://www.passwordstore.org/ --
`aptitude install pass`.

It's very old-school cool.
  --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to RobLa?

[C. Scott Ananian]

It is my understanding that RobLa's departure was not entirely unexpected,
so there were plans in place.  He will be missed!  But Brion and I are
taking up some of his responsibilities for the dev summit, and I understand
there are other folks stepping up to fill other gaps.  I think those of us
who know him well are trying to respect his desire for a bit of a
wikibreak.  If you see some task Rob's been maintaining which seems like it
could use attention, I think the best thing to do right now is to pitch in
and help out for a while.  I strongly suspect he'll be back after a bit,
and I'm hoping he'll find his gardens as well tended as we are able.
  --scott

On Thu, Nov 3, 2016 at 6:48 PM, Florian Schmidt <
florian.schmidt.wel...@t-online.de> wrote:

> Thanks Brian!
>
> That's exactly what this e-mail was for. I was also wondering, what
> happened to him personally, however, it's more than ok and his own decision
> to not say anything about it. However, like Brian wrote already, Robla had
> (as far as I know) a very good standing in the community and it would be a
> good communication about what happened (in any way), instead of "just"
> disabling his account.
>
> However, this e-mail is (and wasn't at any time) to make anyone sad about
> anything, and it makes me sad, that the thread resulted in this feeling for
> some people :(
>
> Best,
> Florian
>
> -Ursprüngliche Nachricht-
> Von: Wikitech-l [mailto:wikitech-l-boun...@lists.wikimedia.org] Im
> Auftrag von bawolff
> Gesendet: Donnerstag, 3. November 2016 21:07
> An: Wikimedia developers 
> Betreff: Re: [Wikitech-l] What happened to RobLa?
>
> Robla obviously deserves his privacy if he wants that. However I think its
> rather natural for people to wonder "What happened" when an employee who is
> as much in the public spotlight as robla is, suddenly and unexpectedly
> leaves.
>
> Obviously just because it is "natural" doesn't mean we deserve answers. I
> think Wes's short email was sufficient in terms of answering the what
> happened question. However, simply putting a global lock on someone's
> account (Which was the state of things before this
> thread) is not good communication and bound to raise eyebrows.
>
> There is also the more pragmatic question of how Robla's responsibilities
> will be transferred. Robla did a lot for MediaWiki, and transitions are
> scary. Transitions with no information are even scarier. How will Robla's
> initiatives be carried forward? Is the architecture committee mature enough
> to continue on purely its own steam without Robla pushing it forward? How
> does this affect the "vision" for MediaWiki (emphasis on MediaWiki, not
> Wikimedia)? Will development of Wikimedia's technology further devolve into
> independent fiefdoms without Robla balancing the architecture needs of
> different groups? All these concerns are in my mind, and I think the minds
> of others. It would be nice to get some reassurance about what is going to
> happen next.
>
> --
> Brian
> [To be 100% clear, posting with my volunteer hat firmly on]
>
> On Thu, Nov 3, 2016 at 4:17 PM, Jonathan Morgan 
> wrote:
> > +1 Petr. This thread makes me sad and uncomfortable.
> >
> > - J
> >
> > On Thu, Nov 3, 2016 at 9:03 AM, Toby Negrin 
> wrote:
> >
> >> Thank you Petr. Well said!
> >> -Toby
> >>
> >> On Thu, Nov 3, 2016 at 2:39 AM, Petr Bena  wrote:
> >>
> >> > Hello,
> >> >
> >> > I don't like to sound like a bad guy, but I really find it quite
> >> > inappropriate to discuss this sort of stuff on a public mailing list.
> >> > This really is a personal matter. If he didn't post any dramatical
> >> > announcement perhaps he wanted to "leave quietly"?
> >> >
> >> > You know, people sometimes do this sort of stuff for very personal
> >> > reasons, they just don't want to discuss with public, and who
> >> > knows, maybe he would return back one day? He wouldn't be first to
> >> > do that :) Or maybe he is going to stick around as a volunteer?
> >> > Let's be optimistic!
> >> >
> >> > If you really want to know the details, you can always ask him
> >> > directly. Fortunately he is not dead, so no need to mourn him. As
> >> > an employee of WMF, he was always very friendly and extremely
> >> > useful, so indeed I /do/ hope he will stay with us, at least as a
> volunteer.
> >> >
> >> > On Thu, Nov 3, 2016 at 10:16 AM, Tilman Bayer
> >> > 
> >> > wrote:
> >> > > On Tue, Nov 1, 2016 at 3:54 AM, Federico Leva (Nemo) <
> >> nemow...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > >> As a reminder on goodbyes, WMF HR used to announce everyone's
> >> > >> last day
> >> > at
> >> > >> https://identi.ca/wikimediaatwork (can't take more than a couple
> >> > >> minutes); then the announcements became monthly, then quarterly;
> >> > >> then
> >> we
> >> > >> lost this courtesy as well https://meta.wikimedia.org/wik
> >> > >> 

Re: [Wikitech-l] 2016W40 ArchCom-RFC meeting: future of magic links

[C. Scott Ananian]

I just have anecdotal knowledge based on
https://phabricator.wikimedia.org/T117165 -- people seem to use ISBNs in
citations often enough to complain when they are 'ed by Visual
Editor, but I don't remember ever having any complaint about s
around the RFC or PMID magic links.
 --scott

On Wed, Oct 5, 2016 at 11:12 AM, Chad  wrote:

> On Tue, Oct 4, 2016 at 11:52 PM Legoktm 
> wrote:
>
> > > 2.  Deprecation strategy for Wikimedia wikis (e.g. Wikipedia)
> >
> > Most of the migration can be done using a bot with some basic regexes,
> > but the key part will be adapting templates to generate links (e.g. ISBN
> > citation templates) instead of relying upon magic link functionality.
> >
> >
> Question: do we have any kinds of numbers yet on how widely these are
> used across WMF projects?
>
> It's info something we could probably get out of either Elasticsearch or
> the
> dumps probably :)
>
> -Chad
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikimedia Developer Summit 2017 discussion

[C. Scott Ananian]

I agree with bawolff, although I also see Quim's point about reopening this
discussion at this particular point in time.

But I think it's an important conversation to have, even if it's not going
to be directly relevant to this year's dev summit.

We used to have a project called "Flow", short for "Work Flow", which
recognized that the WMF projects aren't *just* collaborative content
creation, but also embody a lot of codified *process*.  That seems to be
exactly what Phab is providing for us.

Did we get scared off by our first attempt at solving the work flow
problem?  Or are we eventually going to try to tackle that?
 --scott

On Wed, Sep 28, 2016 at 11:22 AM, Brian Wolff  wrote:

> >
> > It is easy to reply "yes, of course", but looking at the overall list of
> > possible improvements of MediaWiki, I think the Wikimedia movement would
> > prefer the investment of developer time in other areas. These cases are
> > imho too niche from the perspective of writing free encyclopedias,
> > dictionaries, media repositories, etc.
> >
> > In any case, I wouldn't stop anyone willing to work on these features...
> > but neither would I put the organization of the Summit at the expense of
> > any of these features being implemented in MediaWiki. Wikimedia
> Phabricator
> > is a Wikimedia tool that already provides those features, so...
> >
>
> To, me most of these strike me not so much as mediawiki lacking features as
> everyone already being on phabricator and wanting to integrate with those
> people. If you take out the integrate with existing phab content arguments,
> you are basically left with a bunch of features MediaWiki already has.
>
> --
> bawolff
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Community Wishlist" as a Dev Summit topic

[C. Scott Ananian]

On Fri, Sep 23, 2016 at 5:19 PM, Greg Grossmeier  wrote:

> 
> > The suggestion has been raised (
> > https://www.mediawiki.org/wiki/Topic:Tbyqbjcuihhkhtk8) that one of the
> > Topics for the upcoming Developer Summit be the Community Wishlist.
> >
> > It seems to me that the community wishlist is still not completely
> embraced
> > by engineering/devs, perhaps partly because some of the items are
> > impossible, or already on a roadmap, or others have priorities which are
> > out of sync with implementation difficulty.  It is excellent work by the
> > Community Tech team that somehow still feels "not completely integrated".
>
> I'm curious what "completely embraced by engineering" would be? Isn't it
> enough to have a full team structure with management (both engineering
> and product) to be considered embraced? Do we need to do a group hug? ;)
>
> Also, why does it need to be completely embraced by all devs? I know
> many more fully staff projects that are even less embraced (by the
> development community as a whole).


> Also, what is "completely integrated" mean in this context? I don't see
> the tools that they are developing as being oddly non-integrated within
> the workflows they are working with.
>

I'm just relaying a vague sense of reluctance, maybe a bit of NIH.  I don't
see any wishlist items called out on
https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q1_Goals for
instance.  I'm not sure they belong there!  But (to answer your direct
question), it would certainly indicate that the wishlist has been
completely "integrated" and "embraced" by engineering if our
engineering-wide Q2 goals were in explicit alignment with wishlist items.


> tl;dr: I'm trying to figure out why those concerns should demote it?
>

I'm not trying to demote it; I'm arguing that we should be paying more
attention.


> > Perhaps one way to structure a "wishlist" topic at the dev summit would
> be
> > to collaborate to improve the 'status' category of
> > https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey/Results.
> It
> > would be helpful to have an engineering assessment for each wishlist item
> > detailing either:
> >
> >   (a) this is being actively worked on now by WMF staff
> >   (b) this is on a roadmap for (roughly) XYZ date (with a link to the
> > roadmap),
> >   (c) this depends on some other prior work (which is on a roadmap)
> >   (d) this is technically sound but not a priority of the WMF (for
> > , spelled out) so we are eager for community assistance
> >   (e) there is serious disagreement about how to best accomplish this,
> > technically
> >   (f) there is serious disagreement about how to best accomplish this,
> > non-technically (UX, social factors, mission creep, ongoing maintenance,
> > community A disagrees with community B, etc)
> >   (g) this is, in the judgement of engineering, impossible or unwise.
> >
> > It seems that this has been done for the top ten wishlist items, but we
> > could collaborate on filling out details on more of the items.
>
> Would that need to be a DevSummit session? Or a pre-Summit call to
> action/project?
>

I could support either.  At the dev summit it might easier to corral the
disparate teams responsible.  Doing this pre-summit is similar to
continuing the community tech team's current work on the survey -- which is
excellent, but what I'm trying to suggest are ways to make this a project
owned by all of us instead of just "a Community Tech thing".


> > A follow up session could concentrate on items in categories (d) and (e),
> > attempting to resolve roadblocks.  Category (f) would need
> non-engineering
> > participation, perhaps at the next Wikimania.
>
> This sounds like a reasonable use of time at the DevSummit. Most of
> those 'non-technically' aspects are in-fact represented at the DevSummit
> (UX, maintenance concerns, mission creep), and the others that aren't
> decided represented there would, based on past experience, still benefit
> from a conversation with the DevSummit group (social factors, community
> disagreement).
>

Sure.  We've had issues in the past with decision-making in the dev summit
when all the stakeholders weren't present, so I was just trying to narrow
our focus down to the subset of items where enough folks were present that
we could make meaningful decisions that stuck.  Certainly informal
discussions about the other items would be helpful, and perhaps necessary
to set the stage for a future discussion.  But I've heard from the summit
organizing team that they really want to have sessions which conclude with
actionable decisions.  (cf " In previous years a common frustration
has been the disconnect between Summit topics and what happened after in
our actual plans, work, allocation of resources, goals "
https://phabricator.wikimedia.org/E269 ).
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org

[Wikitech-l] "Community Wishlist" as a Dev Summit topic

[C. Scott Ananian]

The suggestion has been raised (
https://www.mediawiki.org/wiki/Topic:Tbyqbjcuihhkhtk8) that one of the
Topics for the upcoming Developer Summit be the Community Wishlist.

It seems to me that the community wishlist is still not completely embraced
by engineering/devs, perhaps partly because some of the items are
impossible, or already on a roadmap, or others have priorities which are
out of sync with implementation difficulty.  It is excellent work by the
Community Tech team that somehow still feels "not completely integrated".

Perhaps one way to structure a "wishlist" topic at the dev summit would be
to collaborate to improve the 'status' category of
https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey/Results.  It
would be helpful to have an engineering assessment for each wishlist item
detailing either:

  (a) this is being actively worked on now by WMF staff
  (b) this is on a roadmap for (roughly) XYZ date (with a link to the
roadmap),
  (c) this depends on some other prior work (which is on a roadmap)
  (d) this is technically sound but not a priority of the WMF (for
, spelled out) so we are eager for community assistance
  (e) there is serious disagreement about how to best accomplish this,
technically
  (f) there is serious disagreement about how to best accomplish this,
non-technically (UX, social factors, mission creep, ongoing maintenance,
community A disagrees with community B, etc)
  (g) this is, in the judgement of engineering, impossible or unwise.

It seems that this has been done for the top ten wishlist items, but we
could collaborate on filling out details on more of the items.

A follow up session could concentrate on items in categories (d) and (e),
attempting to resolve roadblocks.  Category (f) would need non-engineering
participation, perhaps at the next Wikimania.
  --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikimedia Developer Summit 2017: Registration Open

[C. Scott Ananian]

...further, I don't think there's a process (yet) for making proposals for
the dev summit, so I don't think it's possible to answer this question at
this time.

I just assumed that whoever will be evaluating these knows that, and filled
in "not yet!" under "Have you submitted a proposal for an activity at the
Wikimedia Developer Summit?".

Similarly for, "Are there any topics that you would like to see at this
year's event?", there is only an extremely rough draft of these topics at
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Topic_ideas
-- and I think I proposed about half of these.  So I self-cited, but it's
not clear how useful this question will be in general to other attendees at
this point.
  --scott


On Fri, Sep 23, 2016 at 2:14 PM, Ryan Kaldari 
wrote:

> I'm a little confused. Under the "WMF Staff Questions" section of the
> registration it's asking "Why should you be sponsored by WMF to attend the
> Wikimedia Developer Summit?" and it says that "We will only be approving
> WMF staff to attend this event who are planning to actively engage...",
> etc. implying that approval to attend the developer summit is contingent on
> their answer to this question. However, budget owners were already asked to
> fill out flight and lodging approvals for all of their direct reports (for
> both the All Staff and Dev Summit). Is there a second round of vetting that
> will happen at some point?
>
> On Fri, Sep 23, 2016 at 3:37 AM, Bináris  wrote:
>
> > 2016-09-23 12:09 GMT+02:00 Rachel Farrand :
> >
> > > The deadline to request travel sponsorship is Monday October 24th.
> > >
> >
> > And when will the requests be ansered?
> >
> > I am just thinking of travelling (still sot sure how useful it is for me
> > at  the moment and if I can leave my work), but in case of applying I
> would
> > have a trouble.
> >
> > Last year I visited Iran as a tourist, and since then I have been a
> > potential terrorist according to the government of the United States of
> > America. That means I have to apply for a visa 3 months prior to
> travelling
> > (not like other Hungarians who did not visit Iran, and may travel easily
> > for that reason).
> > I think other people may be involved, too.
> >
> >
> > --
> > Bináris
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] WikiDev17 Collaboration topic (Re: Opening up MediaWiki dev summit in January?)

[C. Scott Ananian]

I edited the topic on wiki (
https://www.mediawiki.org/wiki/WikiDev17/Topic_ideas#Collaboration) a
little bit, to separate out the "Technical Collaboration" and "Editor
Collaboration" pieces a bit.  Both are interesting, although I feel like
the target audiences would diverge greatly.  It would be most interesting
to explore how those subtopics could be profitably combined -- why not
dogfood our own tools to do code review with the same mechanisms we use to
review wikitext diffs? -- but I admit that may well be a step too far.
Perhaps simple juxtaposition at the summit will accomplish the desired
cross-fertilization.

It is perhaps unfortunate that we have an existing team named
"Collaboration".  It creates a whiff of toe-stepping and cookie-licking
danger to have a topic eponymous with an existing team.  The page you cite (
https://www.mediawiki.org/wiki/Collaboration) is actually quite useful in
distinguishing the broad idea of "collaboration" from the actual remit (and
great work!) of the currently-constituted team.

Part of why I suggested this summit topic was that I felt that a proper
response to its challenge would of necessity be *cross team* and *whole
org*.  I think our individual parsing, editing, reading, collaboration, etc
teams do a reasonable job setting their own goals.  But what is missing is
the sort of discussion and decision-making that can occur at a broad
summit, where we reconcile (for example):

* improvements to discussion pages (collaboration team)
* real-time collaborative editing (editing team)
* the introduction of "user groups" backing WikiProjects to the core DB
(mythical "core team")
* broading the notion of "a revision" in core DB (mythical "core team")
* real-time reading (reading team)
* social factors in UX to preserve and strengthen and diversify our
community and stop harassment (community engagement, design teams)
* existing workflow mechanisms and projects (collaboration team, mythical
wikiedu team)
* better diff tools (collaboration team + editing team)
* draft namespace/merge tools (i don't think anyone owns this)

So perhaps naming the topic "Collaboration" is as much a mistake as it
would be to name a topic "Editing", "Discovery", "Reading", etc.---although
I don't have another name to propose---since the goal for a summit topic
should be to identify opportunities to solve problems which have proven
difficult to resolve with only smaller-scale collaboration inside our
existing team structures.  To identify fixed points of consensus upon which
we can, Atlas-like, shift the entire organization. :)
  --scott

On Wed, Sep 21, 2016 at 1:53 AM, Rob Lanphier <ro...@wikimedia.org> wrote:

> Scott suggested the following as one of three suggested topic ideas
> for WikiDev17.  The three ideas:
> 1) Collaboration
> 2) Wikitext Maintenance
> 3) Machine Translation
>
> More inline about "1) Collaboration" below:
>
> On Tue, Sep 20, 2016 at 10:05 AM, C. Scott Ananian
> <canan...@wikimedia.org> wrote:
> > *1. *(A unified vision for) *Collaboration*
> >
> >- Real-time collaboration (not just editing, but chatting, curation,
> >patrolling)
> >- WikiProject enhancements: User groups, finding people to work with,
> >making these first class DB concepts
> >- Civility/diversity/inclusiveness, mechanisms to handle/prevent
> >harassment, vandalism, trolling while working together
> >- Real-time reading -- watching edits occur in real time
> >- Integration with WikiEdu
> >- Broadening notion of "an edit" in DB -- multiple contributors,
> >possibly multiple levels of granularity
> >- Tip-toeing toward "draft"/"merge" models of editing
> >- Better diff tools: refreshed non-wikitext UX, timelines, authorship
> >maps, etc.
>
> I've copied this wholesale into the "Collaboration" area on
> [[WikiDev17/Topic ideas]], and quoted it directly here:
> <https://www.mediawiki.org/wiki/Topic:Tbypptt9myumu7q7>
>
> Let's use this thread to focus on this part of Scott's proposal.  A
> lot of these seems in scope for the Wikimedia Collaboration team.
> Does the scope that you're thinking of align with what the team has
> published on their page:
> <https://www.mediawiki.org/wiki/Collaboration>
>
> Rob
> (p.s. please feel free to start separate threads with the other parts
> of Scott's proposal)
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[C. Scott Ananian]

Here are three topic suggestions, cc'ed here in case folks aren't following
the Flow, with an illustrative (but not exhaustive) list of sessions that
could fit under each:

*1. *(A unified vision for) *Collaboration*

   - Real-time collaboration (not just editing, but chatting, curation,
   patrolling)
   - WikiProject enhancements: User groups, finding people to work with,
   making these first class DB concepts
   - Civility/diversity/inclusiveness, mechanisms to handle/prevent
   harassment, vandalism, trolling while working together
   - Real-time reading -- watching edits occur in real time
   - Integration with WikiEdu
   - Broadening notion of "an edit" in DB -- multiple contributors,
   possibly multiple levels of granularity
   - Tip-toeing toward "draft"/"merge" models of editing
   - Better diff tools: refreshed non-wikitext UX, timelines, authorship
   maps, etc.

*2. *Improving* Modular Wikitext Maintenance*

   - Infoboxes from wikidata, categories from wikidata, wikidata in
   commons, oh my!
   - Visual editing of templates, alternative template mechanisms, etc
   - Wikitext 2.0 -- how to shave off the rough edges but still provide a
   text-based power-user editing interface
   - Global pages, Global templates, etc
   - Improving composition of text and media content on the page
   - Moving to a Glossary model for LanguageConverter rules
   - Splitting metadata (categories, page flags, etc) from content in the DB

*3. *(Doubling down on)* Machine Translation*

   - Annotation service to record fine-grained translation correspondences
   between wikis over time (not just at the time of first translation)
   - Suggestion service to suggest new edits to wiki A when translated text
   wiki B is modified (or vice-versa)
   - Refactoring existing language converter pairs as (sometimes trivial)
   translation engines, eg cyrillic-to-latin
   - Building a translation engine in house, training it with translated
   wiki pages, improving it over time, etc
   - Tightly integrating the translation UX for everyone. More: one
   community wearing babel fishes / Less: scattered villagers after the Tower
   of Babel fell.
   - Improving harassment/vandalism/civility/inclusiveness/diversity
   mechanisms to handle these larger cross-cultural communities.
   - i18n of global pages, global templates, etc. May need mechanisms to
   allow translation of comments, for example.

 --scott
-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[C. Scott Ananian]

On Thu, Sep 8, 2016 at 1:57 PM, Rob Lanphier  wrote:

> Good point.  Let's say that we had to pick only one of these questions to
> answer at WikiDev17.  Which would you choose?
>

All of them!  But since no one else bit Rob's bait, let me try to give a
pro/con for each.

* how do we make manipulating the information on our websites easier and
> more useful? (both for humans and computers)
>

Pro: a very broad topic, most reasonable subjects of discussion could fit
under this umbrella.  CF Cite, multilingual support, wikidata-in-commons,
etc.
Con: a very broad topic, would this even be a useful frame?


> * how can we better distribute the information on our websites?
>

Pro: invites radical thought disconnected from monetary/fundraising needs!
How can we get everyone to use our stuff?  We collectively probably haven't
thought hard about this recently.
Con: not related to most of the other proto-topics I've heard floated.
Maybe more of a "strategy" topic than a "dev" topic.


> * how can we help our software development community be more inclusive and
> productive (and attract many more people)?
>

Pro: again, big thoughts, on a topic which deserves attention.  Can drill
down into nitty-gritty like, "why not github?" and "can we make the review
process more friendly?" which are favorite topics for fat-chewing among
developers.
Con: again maybe more "strategy" than "dev"; doesn't help us discuss
wikidata or refactoring the front-end.  Also, why "software development
community more inclusive" and not "editor community more inclusive"?  The
sharp division there might be a real issue.


> * how can we improve the quality of our software, and pay down the
> technical debt we've accumulated over the years?
>

Pro: again, a favorite topic of talk-over-beers among developers.  One
could imagine a whole summit devoted to going through our software stack
component by component, identifying the cruft hidden in each, and making
concrete plans to banish it.
Con: this opinion might be controversial, but my impression is that we're
actually pretty good at low-level refactoring.  There are plenty of things
we are hesitant to change (say, wikitext syntax!), but I don't get the
feeling that the barrier is in engineering.  The problem is mostly a
management one: how can engineering communicate the time spend and value
added by "invisible" maintenance and refactoring; how can we get management
to allocate more dedicates resources to this?  I don't think there's much
technical debate about what to work on, if we had the resources to do so.

* how can we make our websites better learning environments?
>

Pro: another radical idea, and I like radical ideas. ;)
Con: We have wikiversity for this.  Yes, it has its problems.  But wouldn't
this topic be better phrased as, "how can we better support wikiprojects
other than wikipedia?"


> * how can we make our websites better support languages other than English
> (and character sets other than Latin)?
>

Pro: I feel like this was deliberately aimed at me, and I like it. ;)  It
would serve as a concrete frame to recruit new participants from outside
enwiki/SFO.
Con:  Do I have to argue against my own pander?

...

Ok, ok.

Con: the typical attendees of a SFO dev summit are not really the best
folks to discuss non-English/non-Latin issues.  It might be worthwhile
doing as an "educate SFO developers about the issues" training summit, but
if you actually wanted to set goals/make progress you should really host
this topic at a non-North-American Wikimania.

=

Ok, so what have we learned from this?  Even if others have different
opinions about each of Rob's proposed topics, which are the *sort* of
things we'd like the dev summit to be about?  Radical ideas?  Stuff
developers bitch over beers about?  Vague umbrella topics ("make wiki
easier to use") that we can crowd a bunch of stuff under?  Something else
entirely?

  --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[C. Scott Ananian]

On Thu, Sep 1, 2016 at 6:54 PM, bawolff  wrote:

> >
> > * In a big session on services-oriented architectures, a lot of time was
> > spent theorizing about what small wikis who do their hosting on
> > shared-hosting services do, and whether various solutions we were
> proposing
> > would make it easier or harder for these non-WMF users of mediawiki.
> *But
> > none of these users were at the summit.*  So no decisions could
> ultimately
> > be made, as the necessary affected parties were not present.
> >
>
> I don't think that would change, regardless of what we do. Even if we
> have more users at the summit, its not going to be a representative
> sample of every type of user we have. I don't think its reasonable to
> assume that just because a usecase isn't represented at the summit
> that it doesn't exist. Anyone taking the time out of their day to
> travel all the way to a MediaWiki event, is probably a power user, and
> thus this will bias the representation of users at the summit.
>

Quite possibly!  And thus perhaps we should just exclude those sort of
topics from the summit -- if we can't get representative participation, we
shouldn't have a decision-oriented summit session.

Sessions oriented around "learning from our community" might still work --
we could have invited a cross-section of shared hosting users for a
workshop where we gathered info about different hosting providers, walked
them through installs using vagrant (or what-have-you), listened to their
concerns & problems, and worked together with them to figure out what might
work.  At the end, we can't say "this will work for all users of shared
hosts" but at least we can say, "for the 15 people we worked with, here are
the ways they managed to install a services-oriented prototype of
mediawiki" or something like that.

"Community workshop" is an underexplored dev summit format.  But maybe that
would be a better fit at wikimania?
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[C. Scott Ananian]

On Thu, Sep 1, 2016 at 1:12 PM, Brion Vibber  wrote:

> The last couple years we've done a big MediaWiki Dev Summit in January,
> around the time of the Wikimedia Foundation all-hands meeting. Invitations
> have been fairly broad to known developers, but there's a very strong
> feeling that newbies, non-technical people, and in general *the people
> MediaWiki is created and maintained for* are not welcome.
>
> I think we should change this.
>
> I would really like a broader MediaWiki Dev Summit that asks our users to
> participate, and asks "developers" to interact with them to prioritize and
> work on things that really matter to them.
>
> I want template authors, Lua module authors, template users, power editors,
> folks working on the lines of defense for vandalism patrol and copyvio
> checking. I want people with opinions on discussion systems. I want people
> who have been editing for years and have experience with what works and
> what doesn't. I want people who wish they could edit but have a bad
> experience when they try, and want to share that with us so we can help
> make it better.
>

Hear, hear.

To make the discussion concrete, here are some issues I've had at past dev
summits, which may also answer the question "what would these non-devs do?":

* In a big session on services-oriented architectures, a lot of time was
spent theorizing about what small wikis who do their hosting on
shared-hosting services do, and whether various solutions we were proposing
would make it easier or harder for these non-WMF users of mediawiki.  *But
none of these users were at the summit.*  So no decisions could ultimately
be made, as the necessary affected parties were not present.

* I've tried to have conversations about the role of LanguageConverter and
Content Translation at each dev summit.  However, no one was present at the
dev summit who used LanguageConverter on their home wiki, and few folks who
rely on Content Translation routinely.  (Maybe one or two were present, but
not enough to have a reasonable discussion about the future of these
features.)  For better or worse, previous dev summits have had weak
representation from those who are not American users of
projects-other-than-enwiki.  (Again, not that it as 100% American enwiki
users, just that not enough others were present to constitute a reasonable
quorum for discussing issues affecting them.)

* The parsing team has various proposals for improvements to the template
system.  We don't really have a quorum of the "power users" of the wiki
projects who write and use nontrivial templates.

* In general the dev summit is pretty quite about projects other than
wikipedia!  Wikisource/wikibooks/wikitionary/commons/etc have lots of
interesting technical work to be done, which is poorly represented by WMF
employees.

 --scott

ps. I am sympathetic to the idea that this sort of broader conversation
about technical topics might fit better at wikimania.  But the last few
wikimanias have been moving in the opposite direction, to being less
WMF-driven, and I actually thought Esino Lario was a quite nice example of
how that can work.  No one I talked to at Esino Lario felt that "not enough
WMF staff were present" or that they couldn't get WMF answers to their
questions when they needed.  But this trend is opening a gap between WMF
engineering and our user community, which we should try to bridge somehow
or other.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki pingback

[C. Scott Ananian]

Since we have a running debate about services -vs- all-in-one installs -vs-
requiring binary modules, it would be nice to include data points
reflective of these three different hosting scenarios (multiple server,
single server, shared host w/ no ability to install new modules).  To start
the discussion, I might suggest returning a boolean yes/no whether parsoid
is configured (as a proxy for "can services be installed"), a boolean
yes/no for whether mysql is on the same server as php (as a proxy for
"multiple server install"), and a boolean for whether the lua sandbox
extension is installed (as an imperfect proxy for whether binary modules
can be installed), and perhaps a variable reflecting the tidy configuration
(enabled?  if enabled, using the tidy extension or standalone tidy?) as
another insight on binary modules (weakened because i think the tidy
extension is bundled by default with PHP 5).
  --scott


On Fri, Jul 22, 2016 at 10:29 AM, Greg Sabino Mullane 
wrote:

> > The configuration variable that controls this behavior ($wgPingback) will
> > default to false (that is: don't share data). The web installer will
> > display a checkbox for toggling this feature on and off, and it will be
> > checked by default (that is: *do* share data). This ensures (I hope) that
> > no one feels surprised or violated.
>
> Sounds sane, as long as the installer makes it quite clear what it is going
> to be doing.
>
> > - The chosen database backend (e.g., "mysql", "sqlite")
>
> Would love to have DB version information as well (getServerVersion)
>
> Lua version?
>
> > Please chime in if you have any thoughts about this. :)
>
> Many of the wikis I install are on intranets behind heavy firewalls. I'd
> be happy
> to submit this data however if there were an optional method to do so.
>
> --
> Greg Sabino Mullane g...@endpoint.com
> End Point Corporation
> PGP Key: 0x14964AC8
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Reviving SVG client-side rendering task

[C. Scott Ananian]

I'll note that wikipedia also currently uses the SVG systemLanguage option
in a non-standard way which isn't supported by browsers:
https://phabricator.wikimedia.org/T60920

So SVGs which use this feature would have to be blacklisted somehow and
always rendered to PNG.
 --scott

On Fri, May 13, 2016 at 7:58 PM, Gabriel Wicke  wrote:

> Another option might be to piggy-back on the current work towards
> lazy-loaded images [1]. Since this is using JS, it could take into
> account network performance & screen resolutions, in addition to
> browser capabilities. Designing this to degrade gracefully without JS
> might be a bit tricky, though.
>
> Gabriel
>
> [1]: https://phabricator.wikimedia.org/T124390
>
> On Fri, May 13, 2016 at 3:29 PM, Bartosz Dziewoński 
> wrote:
> > On 2016-05-13 22:28, Jon Robson wrote:
> >>
> >> The ResourceLoaderImage module is being used widely to generate SVG
> >> icons with png fallbacks. I'd be interested in seeing if we can use
> >> this in some way for optimising SVGs and removing meta data.
> >
> >
> > I don't know what you have in mind, but please remember that
> > ResourceLoaderImage was not written with security in mind. It has a very
> > simplified version of our usual SVG rendering code, and it assumes that
> any
> > SVG files passed to it is trusted. We traded some caution for some
> > performance. Giving it user-controlled data is going to result in
> security
> > vulnerabilities (at the very least some denial of service ones).
> >
> > --
> > Bartosz Dziewoński
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Gabriel Wicke
> Principal Engineer, Wikimedia Foundation
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Introducing WikiToLearn to developers

[C. Scott Ananian]

On Fri, Nov 27, 2015 at 5:58 PM, Riccardo Iaconelli 
wrote:

> > 2) publish your code in Gerrit:
> > https://www.mediawiki.org/wiki/Git/New_repositories/Requests (you made a
> > Docker extensiona and the skin, right?
>
> We have a few modifications here and there, also on OCG and more standard
> extensions (like Math). We will do more. We're temporarly hosting our
> source
> code on github, while we're reworking with the sysadmins on the
> git.kde.org
> infrastructure, but as a KDE project we will have to move the official
> repos,
> per project policy.
>

Did you complete this move?  Where did your git repositories end up?  I'd
love to look at your patches to OCG and see about upstreaming them.
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Feelings

[C. Scott Ananian]

Keeping the teahouse thread alive...

...for some time I've wanted to prototype some real-time chat and editing
features with the Teahouse folks (eg,
https://www.mediawiki.org/wiki/Extension:TogetherJS) to make the
"conversations" Risker mentions easier/more natural.  If anyone has
suggestions about who to talk to about this, let me know.
 --scott

On Thu, Apr 7, 2016 at 10:58 AM, Aaron Halfaker 
wrote:

> Ori said:
>
> > I would like us to consider the contribution that modifications to the
> user experience make to the
> > interpersonal climate on the wikis.
> >
> > I think that this is important.  Our social experience in computer
> mediated spaces is intertwined with the technologies that manage our
> interactions.  This is certainly true to Wikipedia[1] and I think it is
> true generally[2].  While we may find it easy to discuss the technology and
> social things separately, it is very important that we don't interpret this
> as a real separation.  Our social patterns affect how we choose and design
> our digital technologies and our digital technologies -- in turn -- affect
> our social patterns(for more discussion, see [3]).
>
> J-Mo said:
>
> > If WMF ever supports any additional Teahouse-related development, it
> should
> > be focused on giving more new editors, on more Wikis, access to Teahouses
> > and Teahouse-like tools and resources—rather than doing anything to the
> > Enwiki Teahouse itself, which is doing just fine.
> >
> > But J-Mo, we're literally planning to explore supporting the Teahouse
> with
> more digital technologies right now -- you and I!  E.g. using ORES
>  to identify more good-faith newcomers to route
> to the Teahouse & building a search interface to help newcomers explore
> past questions.  Maybe it's OK because we don't plan to do anything *to*
> the Teahouse, but rather to work *with* the hosts to figure out how to
> build up capacity.   I suspect that, if the technologies we develop are
> able to make the positive social interactions that the Teahouse excels in
> available to more newcomers -- we'll succeed.  And hopefully, if our
> technological investments into the Teahouse fail and somehow make positive,
> human interactions more difficult or otherwise less common, we'll have the
> insight to not deploy them beyond an experiment.
>
> This thread started out as a harmless (and humorous!) joke and it has
> turned into a debate around our values with regards to technologies that we
> intentionally integrate with social behaviors.  I think this is a
> conversation we ought to have, but I'd really like to see us move beyond
> platitudes.  Technology isn't good or bad.  It certainly isn't easy to get
> right, but I believe we can co-evolve our tech and our social structures.
> In a computer mediated environments such as ours, this socio-technical
> co-evolution is our only hope to actually making real progress.
>
> 1. https://meta.wikimedia.org/wiki/Research:The_Rise_and_Decline
> 2. https://en.wikipedia.org/wiki/Sociotechnical_system
> 3. https://www.youtube.com/watch?v=-We4GZbH3Iw#t=34m04s (my "Paramecium
> talk")
>
> -Aaron
>
> On Wed, Apr 6, 2016 at 7:30 PM, Matthew Flaschen 
> wrote:
>
> > On 04/02/2016 09:37 PM, Ori Livneh wrote:
> >
> >> Why am I going on about this? I guess I'm a bit bummed out that the idea
> >> of
> >> designing user interfaces that seek to improve the emotional environment
> >> by
> >> making it easier to be warm and personal to one another is a joke.
> >>
> >
> > For what it's worth, as someone who wasn't involved in that April Fools's
> > "feature", but joke-reviewed it, I did not intend to to discourage any
> > serious efforts to encourage a warm and productive editor community.
> >
> > Matt
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Discovery Weekly Update for the week starting 2016-03-28

[C. Scott Ananian]

Also of note: the completion suggester was used as part of
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2016-03-30/Technology_report
thanks to its evangelization at metrics meeting this week.  Yay!
 --scott

On Fri, Apr 1, 2016 at 5:45 PM, Chris Koerner 
wrote:

> Hello,
> Here is the Discovery department's weekly status update.
>
> * Q4: April - June 2016
> <
> https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q4_Goals#Discovery
> >
> draft goals are available for review.
> * Discovery hosted an IRC office hour
> <
> https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q4_Goals#Discovery
> >
> to give community members a chance to talk to Discovery staff about what
> we're working on.
> * Team members presented at the WMF monthly Metrics meeting
>  (the whole thing is worth
> watching, but Discovery starts at 34:50
> ).
> * Zero result rates with new completion suggester have decreased
> .
> * Wikipedia.org Portal stats are moving in the right direction
> !
> ** Related: Wikipedia Portal clickthrough rates & section usage vary by
> English vs non-English speakers
> .
> * A popularity score based on page views has been loaded into the
> production elasticsearch cluster for use in improving search relevance.
> ** First tests will be run with the completion suggester.
> * Portal A/B testing page
>  created
> and linked to new Portal page
> .
> * A Portal A/B test started  on
> March 22, 2016 and will run until approximately April 5, 2016.
>
> Feedback and suggestions on this weekly update are welcome.
>
> The full update, and archive of past updates, can be found on Mediawiki.org
>
> https://www.mediawiki.org/wiki/Discovery/Status_updates
>
> --
> Yours,
> Chris Koerner
> Community Liaison - Discovery
> Wikimedia Foundation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Feelings

[C. Scott Ananian]

On Fri, Apr 1, 2016 at 3:24 PM, Legoktm  wrote:

> I've written a patch[1] that introduces a new feature to the Thanks
> extension called "feelings". When hovering over a "thank" link, five
> different emoji icons will pop up[2], representing five different
> feelings: happy, love, surprise, anger, and fear. Editors can pick one
> of those options instead of just a plain thanks, to indicate how they
> really feel, which the recipient will see[3].
>


Only one of these options?
  --scott, surprised, happy & a bit fearful

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Let's make parsoid i18n great again

[C. Scott Ananian]

On Fri, Apr 1, 2016 at 3:31 PM, Siebrand Mazeland 
wrote:

> Please give me some time. It's way past beer 'o clock at the Hackathon.
> Tomorrow's another day (JST - Jerusalem Standard Time).
>

Sure, no worries.  I'm waiting for Eran to rebase and document anyway. ;)

Enjoy your beer!
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Let's make parsoid i18n great again

[C. Scott Ananian]

I'm prepared to Be Bold and C+2 https://gerrit.wikimedia.org/r/247914.  In
my view, by establishing a consistent semantics (first alias preferred)
this  empowers the local wikis.  If a particular wiki decides they prefer
to use the English alias in markup, they just have to submit a patch to
reorder the aliases for their particular language.  This is much better
than using some ad hoc mechanism where the choices are arbitrarily made by
the Parsoid maintainers or whoever.

However, the patch needs to be rebased. (Sigh.)  And the new
semanticsordering should really be documented in the code or release notes
somewhere, not just in the gerrit/git history.

Eran, if you can do these two things (and Siebrand doesn't scream "stop"),
I'm happy to C+2.  Arlo is already working on a patch on the Parsoid side
which would switch Parsoid to using the first alias everywhere once the
change to core is deployed.
 --scott

On Fri, Apr 1, 2016 at 2:50 PM, C. Scott Ananian <canan...@wikimedia.org>
wrote:

> Let's get that patch to core +2'ed.  Establishing a consistent preference
> order in core is clearly (IMNSHO) the right thing to do.
>
> Siebrand, could you take a second look at
> https://gerrit.wikimedia.org/r/247914 ?
>  --scott
>
> On Thu, Mar 31, 2016 at 8:32 PM, Arlo Breault <abrea...@wikimedia.org>
> wrote:
>
>>
>> > And there is a consensus for English being bad choice for RTL languages
>> as
>> > it cause mixed directional content which should be avoided. So if we go
>> > with 1 choice, RTL languages should be exception.
>>
>> See https://gerrit.wikimedia.org/r/#/c/280792/
>>
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
>
> --
> (http://cscott.net)
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Let's make parsoid i18n great again

[C. Scott Ananian]

Let's get that patch to core +2'ed.  Establishing a consistent preference
order in core is clearly (IMNSHO) the right thing to do.

Siebrand, could you take a second look at
https://gerrit.wikimedia.org/r/247914 ?
 --scott

On Thu, Mar 31, 2016 at 8:32 PM, Arlo Breault 
wrote:

>
> > And there is a consensus for English being bad choice for RTL languages
> as
> > it cause mixed directional content which should be avoided. So if we go
> > with 1 choice, RTL languages should be exception.
>
> See https://gerrit.wikimedia.org/r/#/c/280792/
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] tags are a usability nightmare for editing on mediawiki.org

[C. Scott Ananian]

Arlo actually implemented the Parsoid side of this:
https://phabricator.wikimedia.org/T50891
We still need VE support: https://phabricator.wikimedia.org/T55974
 --scott

On Fri, Apr 1, 2016 at 1:00 PM, Gregory Varnum <gregory.var...@gmail.com>
wrote:

> I think improving VE to understand translate, and expanding the
> training/documentation for translate are the best next steps. This is also
> a problem on Meta-Wiki.
>
> Removing the translate setup from the pages seems problematic as we're
> trying to make info available in languages beyond just English.
>
> -greg
>
> ___
> Sent from my iPhone - a more detailed response may be sent later.
>
> > On Apr 1, 2016, at 9:49 AM, Brion Vibber <bvib...@wikimedia.org> wrote:
> >
> > So based on notes from Niklas on the bug, it looks like there are two
> main
> > problems:
> >
> > 1) VE treats ... chunks as uneditable extension
> > blobs.
> >
> > 2) Some/many pages use ... incorrectly: spanning
> > paragraphs, markup level boundaries, etc.
> >
> >
> > So problem 1) can be improved in two ways:
> > 1a) Have VE treat ... as more or less like
> plaintext
> > or a div or something
> > 1b) Much fancier sub-document editing
> >
> > Fixing problem 2) means refactoring broken pages in ways that presumably
> > would require at least some of their translations to be re-created.
> >
> > I don't know whether 2) is caused by problems in the translation
> interface,
> > or by people further editing the pages without a detailed understanding
> of
> > how the  and  markup bits work, or a combination
> of
> > both.
> >
> > -- brion
> >
> >
> >
> >> On Fri, Apr 1, 2016 at 7:39 PM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
> >>
> >> On Fri, Apr 1, 2016 at 7:35 PM, C. Scott Ananian <
> canan...@wikimedia.org>
> >> wrote:
> >>
> >>> Proper VE/Parsoid support for the  extension has been wanted
> >>> for
> >>> a while now.  It hasn't made a list of quarterly goals yet.  Help
> wanted,
> >>> certainly.
> >>
> >> This is a HUGELY important dogfooding goal; we need to be able to edit
> the
> >> wiki about our own product!
> >>
> >> Anything we can do to help out?
> >>
> >> -- brion
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] tags are a usability nightmare for editing on mediawiki.org

[C. Scott Ananian]

Proper VE/Parsoid support for the  extension has been wanted for
a while now.  It hasn't made a list of quarterly goals yet.  Help wanted,
certainly.
  --scott
On Apr 1, 2016 12:32 PM, "Brion Vibber"  wrote:

> Lots of pages on mediawiki.org are pretty much uneditable because they're
> strewn with  spanning multiple paragraphs that make VisualEditor
> completely unusable and the source editor very difficult to use.
>
> I'm trying to update documentation, and I'm seriously thinking of regexing
> out all the  stuff so I can actually edit the wiki pages.
>
> This is probably not a good thing.
>
>
> https://phabricator.wikimedia.org/T131516
>
> -- brion
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikipedia in Pig Latin

[C. Scott Ananian]

On a somewhat serious note, I'm a big fan of enabling pig Latin as a
language variant in enwiki.  It would allow out SFO developers to
familiarize themselves with how language converter works, in a script they
can (mostly) read and understand.  It also is *almost but not quite*
reversible (consider "ashway", which could correspond to both "wash" and
"ash"), another property it had in common with many real script
transliterations (often one script will have upper/lowercase distinctions
which are not written in the other script, for example), so it is a useful
test case when thinking about how visual editor (say) ought to work with
variants.

So while you enjoy your igpay atinlay today, spare a thought or two for the
real internationalization issues it resembles.
  --scott
On Apr 1, 2016 11:44 AM, "Ryan Kaldari"  wrote:

> Due to popular demand, I've implemented a patch to dynamically convert all
> Wikipedia content into Pig Latin. Give it a try and let me know what you
> think:
> https://gerrit.wikimedia.org/r/#/c/280927/
>
> Note that this implementation is language agnostic, but it works best with
> Latin scripts. For those of you looking for support for Pig Latin as a
> language variant, try Liangent's patch instead:
> https://gerrit.wikimedia.org/r/#/c/72053/
>
> Cheers,
> Kaldari
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Scope of ArchCom

[C. Scott Ananian]

On Mon, Jan 25, 2016 at 3:16 PM, Rob Lanphier  wrote:

> So: forks welcome!  Any takers?
>

I would love to fork MediaWiki.  Only problem: I'm employed by the WMF.
That would make it a "captive fork" and not actually useful from the
"MediaWiki Foundation" standpoint.

However, I've *also* got far too many projects on my plate already, so
instead I'll briefly outline what my "dream fork" would look like, in the
hopes that someone else will take this ball and run with it:

Goal #1: 100% compatibility with existing MediaWiki content.  This doesn't
(necessarily) mean that the databases need to be the same, or even that it
needs to use wikitext, just that it has good conversion tools "on day 1".

Anti-goal #1: 100% compatibility with MediaWiki *features*.  That is, the
MW core codebase has accumulated a lot of cruft over time.  It shouldn't be
a goal to support *every* possible database backend, special page, or
extension supported by MediaWiki, in fact *not* doing so is exactly why the
fork is needed (the WMF has its hands tied by trying to make every possible
user base happy).

For existing MediaWiki users to migrate this fork, it needs dead-simple
conversion from existing MediaWiki installations (that's goal #1) but it
also needs some compelling new features.  This is actually the easy part, I
think -- WMF is quite tightly constrained by the existing MediaWiki UX, so
there are lots of ideas floating around out there on how to improve things
on the frontend.  There's also quite a lot of enthusiasm for (say)
"github-style" content storage on the backend.  Either one of those things
(or a dozen others) could provide compelling reasons for users to switch.
 "Just a fresh look" could be enough.

The WMF is tightly constrained in ways that a potential fork would not be;
not least by our focus on "Encyclopedic" content.  Prior to employment by
the WMF I worked with MediaWiki for internal documentation within a number
of different technology organizations (heck, WMF itself uses MW this way on
wikitech.wikimedia.org and mediawiki.org) and in most of those contexts a
much more code-focused/version control centric/"markdown"-looking (ie, easy
ways to write  blocks) UX would be compelling.  This is a great
opportunity to make some changes that the WMF can't/won't make in order to
fit a non-encyclopedic need.  (Not to mention non-text media, etc.  There
are products to create libraries on in-house 2d/3d artistic content, used
for games, films, and graphic design; the present MW is almost entirely
unsuited to this.)

Now, for the WMF to eventually migrate to this fork, it would need to
*eventually* be capable of doing all the things the WMF uses MW for in the
Wikipedia projects and others.  But I'd urge any prospective forkers to
think *long term* about this.  Trying to do all WMF things on day one is
probably not the best way to make your fork compelling for some group of
users, and you'll need the support of that group of users in order to grow
and maintain your fork.   Let the WMF worry about migration and
compatibility when it comes to it.  Concentrate first on building something
great.
 --scott

ps. Here's a grab bag of further fork ideas.  None of these are
*necessary*, of course, and some of them in fact may be *unwise*.  But
maybe these will stir the soul and trigger the coding fingers of someone
out there:

* Parsoid provides an excellent substrate for translating existing wikitext
into , whatever you choose that to be.  Parsoid's motto:
"we deal with wikitext, so you don't have to".  We also have good
ContentHandler abstractions in core so that you could treat wikitext as a
"legacy format" in your fork, and use "something better" by default.
 (Markdown is popular with the kids, and there are lots of popular
templating systems these days.)

* Forking MediaWiki means that you can use all the latest PHP features,
right off the bat.  Or else ditch PHP entirely!  It's up to you.  Maybe use
ES2015 JavaScript.  I do suggest sticking to one of the "major" languages
if you wish your fork to have legs... although I'd love to see what an
OCaml-native MediaWiki would look like.  (Or you could code in a hybrid
system, like https://phabricator.wikimedia.org/T114457 , and try to have
the best of multiple worlds.)

* You can change the implementation language but keep the database schema.
Or do the opposite. Lots of ways to avoid reinventing the entire wheel.

* MediaWiki is multilingual in theory, but in practice i18n features
haven't gotten a lot of love in the past decade.  What would a MediaWiki
built around modern machine translation (and spell/grammar correction) look
like?  We tend to build these features around "big data" these days, and WP
is quite a "big data" source.

* I'd love to see more real-time and social features integrated from the
start. But you should probably keep in mind what it takes to create a safe
community as well.  Too many "clean sheet" redesigns foster harassment

Re: [Wikitech-l] Scope of ArchCom

[C. Scott Ananian]

On Mon, Jan 25, 2016 at 4:27 PM, Brad Jorsch (Anomie) <bjor...@wikimedia.org
> wrote:

> On Mon, Jan 25, 2016 at 4:06 PM, C. Scott Ananian <canan...@wikimedia.org>
> wrote:
>
> > On Mon, Jan 25, 2016 at 3:16 PM, Rob Lanphier <ro...@wikimedia.org>
> wrote:
> >
> > > So: forks welcome!  Any takers?
>
> You seem to be assuming that "fork" === "complete rewrite". That's not
> necessarily the case, nor is it necessarily even desirable.
>

I thought I made it clear that a fork could be any combination of things,
up to and including (but not necessarily!) a complete rewrite.  Perhaps I
didn't actually make that clear enough, so I'll state it again:

* You should feel free to use (or not use) as much (or as little) of the
existing mediawiki code as you like.

The important thing (IMO) is that the fork should be easy to migrate to,
and better serve some use case which the existing mediawiki neglects.
 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Scope of ArchCom

[C. Scott Ananian]

On Fri, Jan 22, 2016 at 5:30 PM, Rob Lanphier  wrote:

>  On Fri, Jan 22, 2016 at 2:08 PM, Alex Monk  wrote:
>
> > To clarify - are you saying this ([deploying increasingly excellent
> > software on the Wikimedia production cluster in a consensus-oriented
> > manner]) is the actual current scope of ArchCom, or are you advocating
> for
> > a change in scope?
>
>
> It's my attempt to clarify the scope, but you could argue it's a change.
>
> Ultimately, WMF TechOps has correctly blocked a lot of software making it
> to the Wikimedia cluster that hasn't been through the RFC process, even
> though they themselves weren't entirely clear about the scope.  Wikimedia
> Foundation leadership has an (unfortunately) long history of being unclear
> about the scope.  I share the blame for this.  This is my attempt to
> clarify.
>

Perhaps you could elaborate on the "WMF TechOps" aspect a bit, either here
in email or on the Phab ticket.  It seems that some of the tasks currently
tagged as "RfCs" are actually not ArchCom RfCs (they are
WikiData-related?).  From your description above, it seems there may also
be some not-quite-ArchCom RfCs related to what software gets deployed on
our cluster.

Perhaps we should try to come up with more fine-grained labels for RfCs,
rather than labelling them all "ArchCom RfCs"?   I think there was some
discussion at the dev summit about trying to associate proposals with the
dev summit "working groups", as a way of communicating a broad agenda for
each ArchCom meeting.  Finer-grained RfC labeling might be part and parcel
of this.

  --scott (who isn't opposed to the proposed relabeling in any way, just
thinking perhaps this is an opportunity for better classification)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Peer-to-peer sharing of the content of Wikipedia through WebRTC

[C. Scott Ananian]

This is something we really wanted to have at One Laptop Per Child, and I'm
glad you're looking into it!

In our case, we wanted to be able to provide schools a complete copy of
wikipedia, but could only afford to dedicate about 100MB/student to the
cause (at the time, I think our entire machine had 4GB of storage).  We
ended up using a highly-compressed wikipedia "slice" combining the most
popular articles with a set of articles spidered from a list of educational
topics (so you'd be sure to get articles on all the elements, all the
planets, etc).  But what we really *wanted* to do was to split up the
content among the kid's machines, as you've done, so that between them all
they could have access to a much broader slice of wikipedia.

Now a days, with lower storage costs, it's possible we could give students
a much broader slice of the *text* of wikipedia articles, and still use the
peer-to-peer approach to serve the *media* associated with the articles
(which is much larger than the text content).

The other side of this coin is supporting editability.  We were always
dissatisfied with our read-only slices of the Wiki -- true empowerment
means being able to add and edit content, not just passively consume it.
Of course, collaboratively editing a wikipedia in a peer-to-peer fashion is
a very interesting research project.  I wonder if you think this sort of
thing is in scope for your work.
 --scott


On Sat, Nov 28, 2015 at 1:45 AM, Yeongjin Jang 
wrote:

>
> Hi,
>
> I am Yeongjin Jang, a Ph.D. Student at Georgia Tech.
>
> In our lab (SSLab, https://sslab.gtisc.gatech.edu/),
> we are working on a project called B2BWiki,
> which enables users to share the contents of Wikipedia through WebRTC
> (peer-to-peer sharing).
>
> Website is at here: http://b2bwiki.cc.gatech.edu/
>
> The project aims to help Wikipedia by donating computing resources
> from the community; users can donate their traffic (by P2P communication)
> and storage (indexedDB) to reduce the load of Wikipedia servers.
> For larger organizations, e.g. schools or companies that
> have many local users, they can donate a mirror server
> similar to GNU FTP servers, which can bootstrap peer sharing.
>
>
> Potential benefits that we think of are following.
> 1) Users can easily donate their resources to the community.
> Just visit the website.
>
> 2) Users can get performance benefit if a page is loaded from
> multiple local peers / local mirror (page load time got faster!).
>
> 3) Wikipedia can reduce its server workload, network traffic, etc.
>
> 4) Local network operators can reduce network traffic transit
> (e.g. cost that is caused by delivering the traffic to the outside).
>
>
> While we are working on enhancing the implementation,
> we would like to ask the opinions from actual developers of Wikipedia.
> For example, we want to know whether our direction is correct or not
> (will it actually reduce the load?), or if there are some other concerns
> that we missed, that can potentially prevent this system from
> working as intended. We really want to do somewhat meaningful work
> that actually helps run Wikipedia!
>
> Please feel free to give as any suggestions, comments, etc.
> If you want to express your opinion privately,
> please contact ss...@cc.gatech.edu.
>
> Thanks,
>
> --- Appendix ---
>
> I added some detailed information about B2BWiki in the following.
>
> # Accessing data
> When accessing a page on B2BWiki, the browser will query peers first.
> 1) If there exist peers that hold the contents, peer to peer download
> happens.
> 2) otherwise, if there is no peer, client will download the content
> from the mirror server.
> 3) If mirror server does not have the content, it downloads from
> Wikipedia server (1 access per first download, and update).
>
>
> # Peer lookup
> To enable content lookup for peers,
> we manage a lookup server that holds a page_name-to-peer map.
> A client (a user's browser) can query the list of peers that
> currently hold the content, and select the peer by its freshness
> (has hash/timestamp of the content,
> has top 2 octet of IP address
> (figuring out whether it is local peer or not), etc.
>
>
> # Update, and integrity check
> Mirror server updates its content per each day
> (can be configured to update per each hour, etc).
> Update check is done by using If-Modified-Since header from Wikipedia
> server.
> On retrieving the content from Wikipedia, the mirror server stamps a
> timestamp
> and sha1 checksum, to ensure the freshness of data and its integrity.
> When clients lookup and download the content from the peers,
> client will compare the sha1 checksum of data
> with the checksum from lookup server.
>
> In this settings, users can get older data
> (they can configure how to tolerate the freshness of data,
> e.g. 1day older, 3day, 1 week older, etc.), and
> the integrity is guaranteed by mirror/lookup server.
>
>
> More detailed information can be obtained from the following 

Re: [Wikitech-l] WikiDev '16 working areas

[C. Scott Ananian]

My initial reaction is just that the "user interface presentation" split
feels unnatural.  Almost every other working area requires real input from
the "user interface" side.  Just to pick a single example for each:

* Content Format -- what should the UI for "editing templates" look like?
(T114454)
* Content access & APIs -- what parts of our UI should be "baked in" to our
content APIs?  (ie, default thumbnail size?  stub article cutoffs?)
* Collaboration -- the UX is at least half the task here (T112984)
* Software engineering -- Integrating CSS-preprocessing and templating
mechanisms, say.

The process at [[mw:Wikimedia_Developer_Summit_2016/Scope#Working_Areas]]
states, "We may end up having Phab projects for each of these working
areas, assigning each WikiDev '16 proposal to one primary area. For some
proposals, it may be good to put them in multiple projects, but let's try
to resist that urge."

That's probably a good idea for most of the working groups, but it may be
wise to be more accommodating with "user interface presentation".  We'd
want to ensure that we have adequate "design and UX" representation on all
the proposals which could possibly need it.
  --scott

​
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Git for idiots

[C. Scott Ananian]

This seems like we need to fork the instructions for tools lab -vs- others,
as has been suggested.
 --scott

On Thu, Nov 12, 2015 at 11:43 AM, Purodha Blissenbach <
puro...@blissenbach.org> wrote:

> On WikiMedia Tools Lab neither "git review -s" nor "git review -u" work in
> my tools. They run endlessly until you abort then with a keyboard
> interrupt.
> That is annoying.
> Purodha
>
>
> On 12.11.2015 17:35, C. Scott Ananian wrote:
>
>> Hashar: I think your criticisms of git-review might be out of date.  On my
>> debian system, "sudo apt-get install git-review" was sufficient to install
>> a version of git review which is adequate for all my needs (I have never
>> noticed any of the "known issues" you mention).  And "git review -s" is a
>> one-step way to set up the proper remotes and etc and leave you ready to
>> use it.  I am in the habit of typing "git review -u" since I noticed that
>> sometimes it doesn't update its remotes, but I'm guessing that's more
>> habit
>> than necessity.  None of the other steps you describe seem to be
>> necessary.
>>  --scott
>>
>> On Thu, Nov 12, 2015 at 4:10 AM, Antoine Musso <hashar+...@free.fr>
>> wrote:
>>
>> Le 12/11/2015 03:25, S Page a écrit :
>>> > On Wed, Nov 11, 2015 at 12:50 AM, Petr Bena <benap...@gmail.com>
>>> wrote:
>>> >
>>> >> > Ok, I will try to merge all useful stuff in here:
>>> >> > https://www.mediawiki.org/wiki/Git_for_dummies
>>> >> >
>>> > The problem is these are matters of widely-varying taste and
>>> background.
>>> > When I tried to clean up in early 2013, git experts didn't even agree
>>> on
>>> > whether the gerrit remote should be origin, or whether people should
>>> use
>>> > `git review` at all.
>>>
>>> Hello,
>>>
>>> git-review is a python utility that acts as a thin wrapper around Gerrit
>>> workflow.  It has been written by the OpenStack community which is a
>>> python shop.
>>>
>>> It comes with few issues:
>>>
>>> * the versions that comes with Linux distributions are fairly outdated
>>> and comes with known issues.
>>> * installing using the python package managers is not straightforward
>>> and has several permissions issues depending on your operating system
>>> (some install system wide which require root, others to a user writable
>>> dir etc)
>>> * old versions were pushing to the HEAD of the repo iirc which is
>>> troublesome when you are working on another branch
>>> * some global configuration is needed
>>> * you need a remote named gerrit
>>>
>>> On that last point, git-review 1.26 comes with a new option 'usepushurl'
>>> which makes git-review reuse the origin repo and just set the push url
>>> to the ssh:// url.  The 'gerrit' remote will no more be needed.
>>>
>>> Ie in your ~/.gitconfig :
>>>
>>>[gitreview]
>>> usepushurl = 1
>>>
>>>
>>> What I did until that new version is that all my clones were done with a
>>> remote named 'gerrit' (git clone -o gerrit ).
>>>
>>> Since folks are tired of debugging python stacktraces and incorrect
>>> git-review configuration, some are recommending to use the underlying
>>> Gerrit workflow command:
>>>
>>>  git push origin HEAD:refs/publish/[/]
>>>
>>>
>>>
>>> --
>>> Antoine "hashar" Musso
>>>
>>>
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>>>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC] Balanced templates (was: [RFC] Hygienic templates)

[C. Scott Ananian]

In this repost I forgot to mention that the phab task for `{{#balance}}` is
https://phabricator.wikimedia.org/T114445.

I copied the excellent points made by Gergo into a comment there.
 --scott
​
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC] Balanced templates (was: [RFC] Hygienic templates)

[C. Scott Ananian]

On Wed, Nov 11, 2015 at 8:15 PM, Gergo Tisza <gti...@wikimedia.org> wrote:

> On Tue, Nov 10, 2015 at 1:40 PM, C. Scott Ananian <canan...@wikimedia.org>
> wrote:
>
> >2. `{{#balance:inline}}` would only allow inline (i.e. phrasing)
> content
> >and generate an error if a
> ``/``/`<h*>`/``/``/``/`
> >`/`` tag is seen in the content.
>
>
> Why is  in that list? It's not flow/block content model and filtering it
> out would severely restrict the usefulness of inline templates.
>

That's a good point.  The problem is nested  tags.  So we can either ban
open  tags from the context, or ban  tags from the content.  Or split
things and have {{#balance:link}} vs {{#balance:inline}} or something like
that.  Feedback welcome!  The details of HTML parsing are hairy, and I
wouldn't be surprised if we need slight tweaks to things when we actually
get to the point of implementation.

To be extra specific:  if the context is:

hello, there friend 

and the template is "foo bar", then HTML5 parsing will produce:

hello, there friend foo bar

where the added closing  tag inside the template body prevents "simple
substitution" of the template contents.

   - We just need to emit synthetic `` tokens, the tree
> >builder will take care of closing a tag if necessary or else
> discarding
> > the
> >token.
> >
>
> What if there are multiple levels of unclosed tags?
>

We basically emit enough unclosed tags to close anything which might be
open, and let the tidy phase discard any which are not applicable.

Off-hand, I think the only tag where nesting would be an issue would be
.  So I guess we'll need the Sanitizer to count the open 
tags so we can be sure to emit enough close tags.  Tricky!
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Git for idiots

[C. Scott Ananian]

Hashar: I think your criticisms of git-review might be out of date.  On my
debian system, "sudo apt-get install git-review" was sufficient to install
a version of git review which is adequate for all my needs (I have never
noticed any of the "known issues" you mention).  And "git review -s" is a
one-step way to set up the proper remotes and etc and leave you ready to
use it.  I am in the habit of typing "git review -u" since I noticed that
sometimes it doesn't update its remotes, but I'm guessing that's more habit
than necessity.  None of the other steps you describe seem to be necessary.
 --scott

On Thu, Nov 12, 2015 at 4:10 AM, Antoine Musso  wrote:

> Le 12/11/2015 03:25, S Page a écrit :
> > On Wed, Nov 11, 2015 at 12:50 AM, Petr Bena  wrote:
> >
> >> > Ok, I will try to merge all useful stuff in here:
> >> > https://www.mediawiki.org/wiki/Git_for_dummies
> >> >
> > The problem is these are matters of widely-varying taste and background.
> > When I tried to clean up in early 2013, git experts didn't even agree on
> > whether the gerrit remote should be origin, or whether people should use
> > `git review` at all.
>
> Hello,
>
> git-review is a python utility that acts as a thin wrapper around Gerrit
> workflow.  It has been written by the OpenStack community which is a
> python shop.
>
> It comes with few issues:
>
> * the versions that comes with Linux distributions are fairly outdated
> and comes with known issues.
> * installing using the python package managers is not straightforward
> and has several permissions issues depending on your operating system
> (some install system wide which require root, others to a user writable
> dir etc)
> * old versions were pushing to the HEAD of the repo iirc which is
> troublesome when you are working on another branch
> * some global configuration is needed
> * you need a remote named gerrit
>
> On that last point, git-review 1.26 comes with a new option 'usepushurl'
> which makes git-review reuse the origin repo and just set the push url
> to the ssh:// url.  The 'gerrit' remote will no more be needed.
>
> Ie in your ~/.gitconfig :
>
>[gitreview]
> usepushurl = 1
>
>
> What I did until that new version is that all my clones were done with a
> remote named 'gerrit' (git clone -o gerrit ).
>
> Since folks are tired of debugging python stacktraces and incorrect
> git-review configuration, some are recommending to use the underlying
> Gerrit workflow command:
>
>  git push origin HEAD:refs/publish/[/]
>
>
>
> --
> Antoine "hashar" Musso
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Summit proposal: Turning the Table of Contents into a discrete object

[C. Scott Ananian]

Worth noting that parsoid output does not include the ToC, since it is
expected to be generated after the fact.
  --scott
On Nov 10, 2015 3:20 AM, "Marcin Cieslak"  wrote:

> On 2015-11-10, Isarra Yos  wrote:
> > Hi! I would like to turn the mw ToC into a discrete object within the
> > codebase. Write a ToC class and pull all the random building parts out
> > of the parser and five levels of pageoutput, and make it stop messing up
> > the page caching and stuff. Make this class a thing, separate from the
> > content itself, that can appear on the page or be toggled or messed with
> > or added to or moved or whatever by extensions.
> >
> > I have a proposal about this for the developers summit which is about as
> > specific: https://phabricator.wikimedia.org/T114057
>
> Wow, very good, I would no longer need
> https://www.mediawiki.org/wiki/Extension:DeToc
>
> Lots of nuances probably though.
>
> Saper
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Yandex?

[C. Scott Ananian]

Transcribing from the last Metrics Meeting (
https://www.youtube.com/watch?v=ePV-7nhO-z0 around minute 54, and then
more specifically in response to a question at 55:35): "We have
apertium running... we are going to add the yandex service... we found
out that different languages have different translation tools... so it
really depends on which language pairs ... we are looking for *many*
plugin [backends]."

Cc'ing Santhosh, who was the presenter.

Hope that helps.
 --scott

On Tue, Nov 10, 2015 at 1:47 PM, Legoktm  wrote:
> On 07/02/2015 12:55 PM, Legoktm wrote:
>> On 07/01/2015 06:50 PM, Ricordisamoa wrote:
>>> Il 02/07/2015 03:28, Legoktm ha scritto:
 I noticed: "Yandex coming up soon!" under ContentTranslation. Are there
 more details about what this means?
>>>
>>> https://phabricator.wikimedia.org/T89844 I think
>>
>> Thanks for the pointer. After some more digging, I found
>> .
>>
>> So it appears that ContentTranslation will be contacting a third-party,
>> closed source service? Are users going to be informed that this is the
>> case? What data is being sent?
>>
>
> It appears[1] this has quietly gone ahead without any response here,
> which is disappointing.
>
> [1]
> https://www.mediawiki.org/w/index.php?title=Content_translation/Documentation/FAQ=1935992=1780766
>
> -- Legoktm
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Yandex?

[C. Scott Ananian]

According to my listening of that Metrics meeting, it does seem like
WMF is going to have to pay Yandex for using its service.  But as you
say, that doesn't infect the actual translation text, which goes into
wikipedia and extends the free content available for everyone.
 --scott

On Tue, Nov 10, 2015 at 3:31 PM, John Mark Vandenberg  wrote:
> On Wed, Nov 11, 2015 at 5:47 AM, Legoktm  wrote:
>> On 07/02/2015 12:55 PM, Legoktm wrote:
>>> On 07/01/2015 06:50 PM, Ricordisamoa wrote:
 Il 02/07/2015 03:28, Legoktm ha scritto:
> I noticed: "Yandex coming up soon!" under ContentTranslation. Are there
> more details about what this means?

 https://phabricator.wikimedia.org/T89844 I think
>>>
>>> Thanks for the pointer. After some more digging, I found
>>> .
>>>
>>> So it appears that ContentTranslation will be contacting a third-party,
>>> closed source service? Are users going to be informed that this is the
>>> case? What data is being sent?
>>>
>>
>> It appears[1] this has quietly gone ahead without any response here,
>> which is disappointing.
>>
>> [1]
>> https://www.mediawiki.org/w/index.php?title=Content_translation/Documentation/FAQ=1935992=1780766
>
> As the user is isolated from the communication with Yandex , I don't
> see it as a huge problem.  Using Qualtrics seems to be a much more
> serious problem, and nobody seems to care about that.
>
> Yandex is sort of similar to a "MP4 upload only" support, only without
> the patent concerns.  Relying on it comes at the risk that the service
> stops, but the free content created is not infected.  More likely,
> Yandex will start asking WMF for money, and WMF decides to pay because
> it is 'easier' than terminating using the service.
>
> Anyway, I've added it to
>
> https://meta.wikimedia.org/wiki/Open_source
>
> --
> John Vandenberg
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
(http://cscott.net)

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] RFC: make Parser::getTargetLanguage aware of multilingual wikis

[C. Scott Ananian]

I believe the title language support is for the LanguageConverter
extension.  They used to (ab)use the `{{DISPLAYTITLE:title}}` magic
word in order to use the proper language variant, something like:

`{{DISPLAYTITLE:-{en-us:Color; en-gb:Colour}-}}`

Then support was added to avoid the need for this hack, and just Do
The Right Thing.  I don't know the details, but presumably
`Title::getDisplayLanguage` is part of it.


On Tue, Nov 10, 2015 at 4:00 PM, Brian Wolff  wrote:
> contents on pretty much every other wiki (As an aside, TOC really
> shouldn't split parser cache imo, and that's something I'd like to fix
> at some point, but as it stands, any page with a ToC is split by user
> language)

Then you'll be interested in taking a look at
https://phabricator.wikimedia.org/T114057
 --scott

-- 
(http://cscott.net)

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [RFC] Balanced templates (was: [RFC] Hygienic templates)

[C. Scott Ananian]

The RFC proposal for "hygienic templates" got improved a bunch -- and
renamed to "balanced templates" -- during the parsing team off-site.  It
seems like it's worth resending this to the list.  Comments on the updated
draft welcome!



As described in my Wikimania 2015 talk

 (starting at slide 27
),
there are a number of reasons to mark certain templates as "balanced".
Foremost among them: to allow high-performance incremental update of page
contents after templates are modified, and to allow safe editing of
template uses using HTML-based tools such as Visual Editor or jsapi
.

This means (roughly) that the output of the template is a complete
DocumentFragment
: every
open tag is closed and there are no nodes which the HTML adoption agency
algorithm
 will
reorder.  (More precise details below.)

Template balance is enforced: tags are closed or removed as necessary to
ensure that the output satisfies the necessary constraints, regardless of
the values of the template arguments or how child templates are expanded.
You can imagine this as running tidy (or something like it
) on the template output before
it is inserted into the document; but see below for the actual
implementation.

The primary benefit of balanced templates is allowing efficient update of
articles by doing substring substitution for template bodies, without
having to expand all templates to wikitext and reparse from scratch.  It
also guarantees that the template (and surrounding content) will be
editable in Visual Editor; mistakes in template arguments won't "leak out"
and prevent editing of surrounding content.

***Wikitext Syntax***

After some bikeshedding, we decided that balance should be an "opt-in"
property of templates, indicated by adding a `{{#balance:TYPE}}` marker to
the content.  This syntax leverages the existing "parser function" syntax,
and allows for different types of balance to be named where `TYPE` is.

We propose three forms of balance, of which only the first is likely to be
implemented initially.  Other balancing modes would provide safety in
different HTML-parsing contexts.  We've named two below; more might be
added in the future if there is need.


   1. `{{#balance:block}}` would close any open ``/``/``/``
   tags in the article preceding the template insertion site.  In the template
   content all tags left open at the end will be closed, but there is no other
   restriction.  This is similar to how block-level tags work in HTML 5. This
   is useful for navboxes and other "block" content.
   2. `{{#balance:inline}}` would only allow inline (i.e. phrasing) content
   and generate an error if a ``/``/``/``/``/``/`
   `/`` tag is seen in the content.  But because of this, it //*can*//
   be used inside a block-level context without closing active ``/``/`
   `/`` in the article (as `{{#balance:block}}` would).  This is
   useful for simple plain text templates, e.g. age calculation.
   3. `{{#balance:table}}` would close ``/``/`` but would allow
   insertion inside `` and allow ``/`` tags in the content.
(There might be some other content restrictions to prevent fostering.)

We expect `{{#balance:block}}` to be most useful for the large-ish
templates whose efficient replacement would make the most impact on
performance, and so we propose `{{#balance:}}` as a possible shorthand for `
{{#balance:block}}`.  (The current wikitext grammar does not allow `
{{#balance}}`, since the trailing colon is required in parser function
names, but if desired we could probably accommodate that abbreviation as
well without too much pain.)

Violations of content restrictions (ie, a `` tag in a `
{{#balance:inline}}` template) would be errors, but how these errors would
be conveyed is an orthogonal issue.  Some options for error reporting
include ugly bold text visible to readers (like `{{cite}}`), wikilint-like
reports, or inclusion in `[[Category:Balance Errors]]`.  Note that errors
might not appear immediately: they may only occur when some other included
template is edited to newly produce disallowed content, or only when
certain values are passed as template arguments.

***Implementation***

Implementation is slightly different in the PHP parser and in Parsoid.
Incremental parsing/update would necessarily not be done in the PHP parser,
but it does need to enforce equivalent content model constraints for
consistency.

PHP parser implementation strategy:

   - When a template with `{{#balance}}` is expanded, add a marker to the
   start of its output.
   - In the Sanitizer leave 

Re: [Wikitech-l] Parsoid still doesn't love me

[C. Scott Ananian]

On Mon, Nov 9, 2015 at 1:37 PM, Petr Bena  wrote:

> Do you really want to say that reading from disk is faster than
> processing the text using CPU? I don't know how complex syntax of mw
> actually is, but C++ compilers are probably much faster than parsoid,
> if that's true. And these are very slow.
>
> What takes so much CPU time in turning wikitext into html? Sounds like
> JS wasn't a best choice here.
>

More fundamentally, the parsing task involves recursive expansion of
templates and image information queries, and popular wikipedia articles can
involve hundreds of templates and image queries.  Caching the result of
parsing allows us to avoid repeating these nested queries, which are a
major contributor to parse time.

One of the benefits of the Parsoid DOM representation[*] is that it will
allow in-place update of templates and image information, so that updating
pages after a change can be by simple substitution and *without* repeating
the actual "parse wikitext" step.
  --scott
[*] This actually requires some tweaks to the wikitext of some popular
templates; https://phabricator.wikimedia.org/T114445 is a decent summary of
the work (although be sure to read to the end of the comments, there's
significant stuff there which I haven't editing into the top-level task
description yet).

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Forking, branching, merging, and drafts on Wikipedia

[C. Scott Ananian]

Apologies for the summit proposal reading like a manifesto.  Drafts
are a big use case, as is offline editing.  Flagged revisions might
use this as well. As a feature request it dates back to the dark days
of the wiki.  It certainly is an enabler for a lot of different
editing/revision/collaboration models that people have proposed over
the years.
 --scott

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

Let's not let this discussion sidetrack into "shared hosting vs VMs (vs
docker?)" --- there's another phabricator ticket and summit topic for that (
https://phabricator.wikimedia.org/T87774 and
https://phabricator.wikimedia.org/T113210.

I'd prefer to have discussion in *this* particular task/thread concentrate
on:

* Hey, we can have JavaScript and PHP in the same packaging system.  What
cool things might that enable?

* Hey, we can have JavaScript and PHP running together in the same server.
Perhaps some persistence-related issues with PHP can be made easier?

* Hey, we can actually write *extensions for mediawiki-core* in JavaScript
(or CoffeeScript, or...) now.  Or run PHP code inside Parsoid.  How could
we use that?  (Could it grow developer communities?)

* How are parser extensions (like, say, WikiHiero, but there are lots of
them) going to be managed in the long term?  There are three separate
codebases to hook right now.  An extension like  might eventually
need to hook the image thumbnail service, too.  Do we have a plan?

And the pro/anti-npm and pro/anti-docker and pro/anti-VM discussion can go
into one of those other tasks.  Thanks.

 --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Forking, branching, merging, and drafts on Wikipedia

[C. Scott Ananian]

There is a proposal for the upcoming Mediawiki Dev Summit to get us
"unstuck" on support for non-linear revision histories in Wikipedia.  This
would include support for "saved drafts" of wikipedia edits and offline
editing support, as well as a more permissive/friendly 'fork first' model
of article collaboration.

I outlined some proposed summit goals for the topic, but it needs a bit of
help if it is going to make the cut for inclusion.  I hope interested folks
will weigh in with some comments on
https://phabricator.wikimedia.org/T113004 --- perhaps suggesting specific
"next step" projects, for instance.

Thanks for your help.
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

Tyler: I hear you.  I'm not sure it's a good idea, either -- especially not
for core extensions used in production.

But it does perhaps allow some expansion of our developer community on the
fringes, and makes writing extensions possible for a larger set of people?
And perhaps there are some cool things written in JavaScript which the
extended community could more easily hook up to MediaWiki using `php-embed`.

I'm not sure that there are.  I'm just opening up the discussion to see if
anyone pipes up with, "oh, yeah, I've always wanted to do XYZ!".

Greg: I agree re: premature stifling of discussion.  I'm just saying that
"high-level" conversation is already happening elsewhere, and it's more
productive there.  I started *this* particular thread trying to elicit
discussion more narrowly focused on the thing I've just built.
  --scott

On Fri, Nov 6, 2015 at 2:30 PM, Tyler Romeo <tylerro...@gmail.com> wrote:

> I would very, *very* much prefer to not have MediaWiki core extensions
> written in JavaScript. Even beyond my criticisms of JavaScript as a
> language, I feel like that just unnecessarily introduces complexity. The
> purpose of this wrapper is to combine separate micro-services that would
> otherwise be run in separate VMs / servers / etc. so that it can easily be
> run in a hosting setup.
>
> Otherwise, I'm interested in what implications this will have, especially
> for making MediaWiki easier to install and use, which would be awesome.
>
> --
> Tyler Romeo
> https://parent5446.nyc
> 0x405D34A7C86B42DF
>
> From: C. Scott Ananian <canan...@wikimedia.org> <canan...@wikimedia.org>
> Reply: Wikimedia developers <wikitech-l@lists.wikimedia.org>
> <wikitech-l@lists.wikimedia.org>
> Date: November 6, 2015 at 14:14:13
> To: Wikimedia developers <wikitech-l@lists.wikimedia.org>
> <wikitech-l@lists.wikimedia.org>
> Subject:  Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`
>
> Let's not let this discussion sidetrack into "shared hosting vs VMs (vs
> docker?)" --- there's another phabricator ticket and summit topic for that
> (
> https://phabricator.wikimedia.org/T87774 and
> https://phabricator.wikimedia.org/T113210.
>
> I'd prefer to have discussion in *this* particular task/thread concentrate
> on:
>
> * Hey, we can have JavaScript and PHP in the same packaging system. What
> cool things might that enable?
>
> * Hey, we can have JavaScript and PHP running together in the same server.
> Perhaps some persistence-related issues with PHP can be made easier?
>
> * Hey, we can actually write *extensions for mediawiki-core* in JavaScript
> (or CoffeeScript, or...) now. Or run PHP code inside Parsoid. How could
> we use that? (Could it grow developer communities?)
>
> * How are parser extensions (like, say, WikiHiero, but there are lots of
> them) going to be managed in the long term? There are three separate
> codebases to hook right now. An extension like  might eventually
> need to hook the image thumbnail service, too. Do we have a plan?
>
> And the pro/anti-npm and pro/anti-docker and pro/anti-VM discussion can go
> into one of those other tasks. Thanks.
>
> --scott
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>


-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

On Fri, Nov 6, 2015 at 4:12 PM, Brion Vibber <bvib...@wikimedia.org> wrote:

> On Fri, Nov 6, 2015 at 11:13 AM, C. Scott Ananian <canan...@wikimedia.org>
> wrote:
> > * Hey, we can have JavaScript and PHP running together in the same
> server.
> > Perhaps some persistence-related issues with PHP can be made easier?
> >
>
> We probably wouldn't want to break the PHP execution context concept that
> requests are self-contained (and failing out is reasonably safe). But you
> could for instance route sessions or cache data through the containing node
> server instead of on the filesystem or a separate memcache/etc service...
>

Right, exactly.  I'm currently running Opcache and APCu inside the embedded
PHP which are going to some lengths to offer persistent caches.  I'm not an
expert on PHP architecture; I suspect there are other places in mediawiki
where we are similarly jumping through hoops.  Perhaps these could be
simplified, at least for certain users.


> > * Hey, we can actually write *extensions for mediawiki-core* in
> JavaScript
> > (or CoffeeScript, or...) now.  Or run PHP code inside Parsoid.  How could
> > we use that?  (Could it grow developer communities?)
> >
>
> I'm uncertain about the desirability of general direct JS<->PHP sync call
> bridging, in that relying on it would _require_ this particular node+PHP
> distribution. I'd prefer loose enough coupling that the JS engine can be
> local or remote, and the PHP engine can be either Zend or HHVM, etc.
>

I expect that I can port php-embed to PHP 7 and/or HHVM without too much
trouble, if interest warrants.  And I already support quite a large number
of different node versions, from 2.4.0 to 5.0.0.  And there are some
interesting other alternative implementations that could export the same
interface but use RPC to bridge node and PHP, see for instance
https://github.com/bergie/dnode-php.  Even the sync/async distinction can
be bridged; if you look at the underlying implementation for php-embed all
communication is done via async message passing between the threads.  We
just "stop and wait" for certain replies to emulate sync calls (in
particular for PHP, which prefers it that way).


> Of course there are interesting possibilities like using JS as a template
> module extension language in place of / addition to Lua. A general warning:
> as I understand the php-embed bridge, JS-side code would a) have full
> rights to the system within the user the daemon runs as, and b)
> exiting/failing out of node would kill the entire daemon.
>

There is sandboxing within v8, so your warning is not accurate.

And in fact, the "mirror image" project is the PHP extension v8js, which I
believe Tim started and I worked on for a while before attempting
node-php-embed.  It also uses the native v8 sandboxing facilities.


> PHP-inside-Parsoid might be interesting for some kinds of extensions, but
> I'm not sure whether it's better to rig that up versus using looser
> coupling where we make an internal HTTP call over to the PHP MediaWiki
> side.
>

Yup.  That's what we essentially already do: after starting to implement
the template engine in Parsoid, it was scrapped and the entire templating
engine is implemented by calling over to PHP to allow it to expand
templates.  And whenever we want more information about the expansion, we
implement it in PHP.

But that's essentially the genesis of the "mediawiki as a collection of
services" idea -- once you start doing this, you find all sorts of bits of
crufty complex PHP code which you'd rather not try to reimplement.  First
templates, then image thumbnailing, next who knows, probably the skin.  One
day they might all be spun out as separate services with internal HTTP
calls between them.

I'm just providing a PoC that lets you ask questions about potential
alternatives.  I welcome the discussion.

> * How are parser extensions (like, say, WikiHiero, but there are lots of
> > them) going to be managed in the long term?  There are three separate
> > codebases to hook right now.  An extension like  might
> eventually
> > need to hook the image thumbnail service, too.  Do we have a plan?
> >
>
> This probably deserves its own thread!
>

Yeah.

Ideally you should only have to write one implementation, and it should be
> self-contained or access the container via a limited API.
>
> I'm not really sure I grasp how Parsoid handles tag hook extensions at the
> moment, actually... can anyone fill in some details?
>

It doesn't basically.  It just asks PHP to do the expansion for it, and
then wraps the whole thing in an element warning VE not to touch it.

Except for citations.

On our roadmap for this quarter we have a task to write a "proper"
extension interface, and then use it to refa

Re: [Wikitech-l] Parsoid still doesn't love me

[C. Scott Ananian]

I think your subject line should have been "RESTBase doesn't love me"?
 --scott
​
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

On Fri, Nov 6, 2015 at 4:52 PM, Daniel Friesen 
wrote:

> That all being said. I still think the original rationale for picking
> lua (more sandboxing controls including execution limits based on steps
> in lua rather than varying execution time) is still valid.
>

It's not, actually.  It may have been at the time.  But v8 now has both
time and memory limits and fine-grained counters for various events with
callbacks and all sorts of crazy sandbox-y things.  See
https://github.com/phpv8/v8js/blob/master/v8js_timer.cc (although I think
the latest v8 actually has even more direct ways of enforcing these limits.)
 --scott, who'd like to get some more work done on Scribunto/JS at some
point.

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MW support for Composer equivalent for JavaScript packages

[C. Scott Ananian]

Note that as part of https://phabricator.wikimedia.org/T114457 I'm
experimenting with distributing mediawiki-core itself as an npm-installable
package.  It ought to be possible to package up extensions, including
Parsoid and VE, using npm as well.
 --scott

On Thu, Nov 5, 2015 at 11:32 AM, Trevor Parscal 
wrote:

> The flat approach to NPM is a game changer for us, and a Bower killer.
>
> Timo? Had a lot of insight at the time, I'd like to see him be invoked in
> this decision. Any thoughts Timo?
>
> - Trevor
>
> On Thursday, November 5, 2015, Jon Robson  wrote:
>
> > It's been a year now, over 3-6 months. Volker can this be given a
> > priority in the next frontend standards meeting. I think the lack of
> > any standard is extremely damaging to the project.
> >
> > On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis  > > wrote:
> > > On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner
> > > > wrote:
> > >> Hey,
> > >>
> > >> I just wanted to check in about the status of enabling JavaScript
> > package
> > >> management usage in MediaWiki. I am basically talking about an
> > equivalent
> > >> for JS to what we have with Composer for PHP.
> > >>
> > >> Real-world example:
> > >>   The "data-values/value-view" package[0] is defining
> > >> "jquery.event.special.eachchange.js":
> > >> ValueView/lib/jquery.event/jquery.event.special.eachchange.js
> > >>
> > >>   Now, recently I needed the same functionality in one of my
> > extensions, so
> > >> I just copied it over. [1]
> > >>
> > >> I know that this is the worst way one could do this, but as far as I
> can
> > >> see we don't have that much of a choice right now. Here are the
> > alternative
> > >> options I can see:
> > >>
> > >> Moving "jquery.event.special.eachchange.js" out of the
> > >> "data-values/value-view" package into its own "WMDE/jquery-eachchange"
> > >> package...
> > >>
> > >> 1. ... and using it in my extension via composer.
> > >>   + pro: two or more extensions or other packages requiring this
> package
> > >> will still result in having only one MW-wide installation..
> > >>   - con: requires MW specific code which is actually not related to
> the
> > >> MW-independent package to feed the resource loader.
> > >>   - con: using Composer to manage pure JavaScript packages! Uuuh,
> ugly!
> > >>
> > >> 2. ... and having a build step in other packages using the package,
> > pulling
> > >> the "WMDE/jquery-eachchange" somewhere into the file structure of the
> > >> packages/extensions using it.
> > >>   + pro: don't need to abuse composer, we can use "npm", "Bower" or
> any
> > >> other arbitrary JS package manager here.
> > >>   - con: got to tell resource loader somehow... (didn't think so much
> > about
> > >> that yet)
> > >>   - con: if more than one extensions or other packages require this
> > package
> > >> we still end up with the same code twice or more often in one MW
> > >> installation.
> > >>
> > >> 3. Combining 1 and 2: Start with 2, using a JS package manager. Then
> > going
> > >> to 1, creating a composer package and pulling the
> > "WMDE/jquery-eachchange"
> > >> package in via some build script.
> > >>   + pro: The two pros from 1 + 2
> > >>   + pro: ^^
> > >>   - con: still got to tell resource loader somehow...
> > >>   - con: Overhead; We now create two packages where the Composer one
> is
> > >> just a bridge to the MW-world, still polluting packagist.org. Still
> > kind of
> > >> ugly and more effort for publishing a package and therefore
> potentially
> > >> scaring programmers away from doing so since they've got better things
> > to
> > >> do than doing work that could be automated.
> > >>
> > >> I have not seen Approach 2 and 3 yet. Though I could imagine that the
> > >> VisualEditor team has used something like that.
> > >> Approach 1 is the way the "data-values/value-view" package itself is
> > being
> > >> handled. And that package should actually be a MW independent pure JS
> > >> package but right now it contains MW specific code and uses composer
> for
> > >> distribution!
> > >> There is still another option but that had to be properly implemented:
> > >>
> > >> 4. Choose one native JS package manager for now and go with it (add
> > support
> > >> for others later perhaps). Integrate it properly with MW (resource
> > loader
> > >> to begin with), document how to use it and finally distribute JS code
> > >> coming from the MW world but useful for other projects in a way where
> it
> > >> can actually be used in a non-MW context.
> > >>
> > >> This has already been bugging me when working on Wikidata. Now I'd
> like
> > to
> > >> reuse some of the code I have written there without spending hours and
> > >> hours with option 3 because there should be support for option 4
> rather
> > >> sooner or later.
> > >> So I am wondering; Does anyone have any thoughts, any alternatives
> > perhaps
> > 

Re: [Wikitech-l] mediawiki-extension cli extension installer/updater released

[C. Scott Ananian]

Do you think this would work with
https://github.com/cscott/node-mediawiki-express ?
 --scott

On Thu, Oct 8, 2015 at 4:12 AM, Brion Vibber  wrote:

> Nice!
>
> -- brion
>
> On Thursday, October 8, 2015, Daniel Friesen  > wrote:
>
> > As part of a side project in my job at Redwerks[1], I've released an
> > early version of a mediawiki-extension cli tool.
> >
> >   https://github.com/redwerks/mediawiki-extension-command#readme
> >
> > The tool requires Node.js (in addition to git and php to run clones and
> > composer).
> > It can be installed with `sudo npm install -g mediawiki-extension` and
> > then `mediawiki-extension setup`.
> >
> >
> > The command can download and upgrade any extension we have in Gerrit.
> > Extensions using composer will automatically be installed via composer
> > otherwise it'll be cloned from git. If available, release tags (like
> > "1.0.0" or "v3.0.0") will be used, otherwise master will be used.
> >
> > You'll still need to require and configure the extension yourself. But
> > this is supposed to greatly simplify acquiring and bulk-updating
> > extensions via the cli.
> >
> > Some examples of use.
> >
> > Clone ParserFunctions from git.
> >   $ mediawiki-extension download ParserFunctions
> >
> > Install SemanticMediaWiki and SemanticForms with composer.
> >   $ mediawiki-extension download SemanticMediaWiki SemanticForms
> >
> > Clone Widgets with from git and checkout the most recent version tag.
> >   $ mediawiki-extension download Widgets
> >
> > Update all your MediaWiki extensions:
> >   $ mediawiki-extension update --all
> >
> > Switch an extension cloned from git master to the REL branch for the
> > installed version of MediaWiki.
> >   $ mediawiki-extension switch ParserFunctions git-rel
> >
> >
> > [1] http://redwerks.org/
> >
> > --
> > ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

Architecturally it may be desirable to factor our codebase into multiple
independent services with clear APIs, but small wikis would clearly like a
"single server" installation with all of the services running under one
roof, as it were. Some options previously proposed have involved VM
containers that bundle PHP, Node, MediaWiki and all required services into
a preconfigured full system image. (T87774
)

This summit topic/RFC proposes an alternative: tightly integrating PHP/HHVM
with a persistent server process running under node.js. The central service
bundles together multiple independent services, written in either PHP or
JavaScript, and coordinates their configurations. Running a
wiki-with-services can be done on a shared node.js host like Heroku.

This is not intended as a production configuration for large wikis -- in
those cases having separate server farms for PHP, PHP services, and
JavaScript services is best: that independence is indeed the reason why
refactoring into services is desirable. But integrating the services into a
single process allows for hassle-free configuration and maintenance of
small wikis.

A proof-of-concept has been built. The node package php-embed
 embeds PHP 5.6.14 into a node.js
(>= 2.4.0) process, with bidirectional property and method access between
PHP and node. The package mediawiki-express
 uses this to embed
MediaWiki into an express.js  HTTP server. (Other
HTTP server frameworks could equally well be used.)  A hook in the `
LocalSettings.php` allows you to configure the mediawiki instance in
JavaScript.

A bit of further hacking would allow you to fully configure the MediaWiki
instance (in either PHP or JavaScript) and to dispatch to Parsoid (running
in the same process).

*SUMMIT GOALS / FOR DISCUSSION*


   - Determine whether this technology (or something similar) might be an
   acceptable alternative for small sites which are currently using shared
   hosting.  See T113210  for
   related discussion.
   - Identify and address technical roadblocks to deploying modular
   single-server wikis (see below).
   - Discuss methods for deploying complex wikitext extensions.  For
   example, the WikiHiero
    extension would
   ideally be distributed with (a) PHP code hooking mediawiki core, (b)
   client-side JavaScript extending Visual Editor, and (c) server-side
   JavaScript extending Parsoid.  Can these be distributed as a single
   integrated bundle?


*TECHNICAL CHALLENGES*


   - Certain pieces of our code are hardwired to specific directories
   underneath mediawiki-core code.  This complicates efforts to run mediawiki
   from a "clean tree", and to distribute piece of mediawiki separately.  In
   particular:
   - It would be better if the `vendor` directory could (optionally) live
  outside the core mediawiki tree, so it could be distributed
separately from
  the main codebase, and allow for alternative package structures.
  - Extensions and skins would benefit from allowing a "path-like" list
  of directories, rather than a single location underneath the
core mediawiki
  tree.  Extensions/skins could be distributed as separate packages, with a
  simple hook to add their locations to the search path.
  - Tim Starling has suggested that when running in single-server mode,
   some internal APIs (for example, between mediawiki and Parsoid) would be
   better exposed as unix sockets on the filesystem, rather than as internet
   domain sockets bound to localhost.  For one, this would be more "secure by
   default" and avoid inadvertent exposure of internal service APIs.
   - It would be best to define a standardized mechanism for "services" to
   declare themselves & be connected & configured.  This may mean standard ro
   utes on a single-server install (`/w` and `/wiki` for core, `/parsoid`
   for parsoid, `/thumb` for the thumbnailer service, etc), standard ports
   for each service (with their own http servers), or else standard locations
   for unix sockets.
   - Can we leverage some of the various efforts to bridge composer and npm
   (for example ), so we
   don't end up with incompatible packaging?

Phabricator ticket: https://phabricator.wikimedia.org/T114457

Download the code for mediawiki-express and play with it a bit and let's
discuss!
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MW support for Composer equivalent for JavaScript packages

[C. Scott Ananian]

Two other interesting pieces:

1. http://requirejs.org/ is still the goal-standard for async browser-type
loading, AFAIK, and there are good packages (`npm install requirejs`) that
allow interoperability with the "npm style".

2. The recently-completed ES6 spec contains a module format.  You can
already use it via compatibility thunks from many existing module systems.
It may well see increased use, especially on the web as browsers implement
the spec (which is happening quickly).  There is resistance in the node
community to adopting ES6 modules, but it may be that we are at an
inflection point and ES6 will eventually win out.
  --scott

On Thu, Nov 5, 2015 at 3:24 PM, Daniel Friesen 
wrote:

> As someone who now works in js/node primarily I might be able to add a
> little outside insight.
>
> IMHO bower was probably killed far before npm@3 came out.
>
> - npm has always been more advanced
> - bower's main isn't as reliable as it is in npm. Even if modules aren't
> forgetting to set it as much as they used to I still see hard to deal
> with things like modules that include .sass sources in it.
> - bower's distribution model is actually very problematic for upstream
> developers. bower clones git repositories directly expecting to find
> built/compiled js and has no local build hooks. Few major js libraries
> work in classic js/css sources anymore, instead having some required
> build step. Bower makes a complete mess of this. Requiring the build
> result to be committed with each change, an automated build bot, a
> second git repo for build results only, or the upstream dev to just not
> care about bower.
> - Thanks to the rise of browserify and the like more and more
> client-only libraries have moved to npm despite people traditionally
> thinking of it as being for nodejs modules. Most client only libraries
> now exist in npm. And if you wave the name browserify around you can get
> just about any js library to start distributing to npm.
>   - This has actually gone so far that once when I added a contribution
> to a library I noticed that they were actually forgetting to keep their
> bower.json up to date.
>
> npm@3 is also probably not as important as it's being made out here.
>
> npm@3 still doesn't guarantee a tree will be 100% flat. Most of npm@3's
> changes fix small quirks in front-end usage and stability issues with npm.
>
> The 'major' change of 'flatness' is really that when installing `a` that
> depends on `b`, `b` will be hoisted up to the same level as `a`, if
> possible, instead of always being installed under `a`. npm@2 did have
> some annoying quirks during development/updating that could leave a
> duplicate module until you recreated your node_modules tree. And there
> was the side case of installing two modules that both depended on
> something you did not depend on. But that could be dealt with by either
> adding it as a dep yourself or running dedupe.
>
> 
>
> The bad news is that while more and more libraries are moving to npm the
> underlying reason for many being on npm is for the many users using
> tools like browserify and webpack. So the assumption of many libraries
> is that when you use npm for client-side libraries is you are using it
> in a node-style/CommonJS way where require is available other npm
> dependencies can be used through it and you're not including things in
> the traditional way of libraries being run one after another on the page
> in the global scope like in ResourceLoader.
>
> It's probably going to get harder and harder to reconcile deps shared
> between extensions and/or use library code itself without having node
> installed on servers. For that matter half the advancements in the css
> space probably won't be making their way to php either.
>
> Though I do have some hints to things that might open up some
> possibilities.
>
> Firstly browserify can build standalone versions of libraries. It's
> possible you could make tweaks to that pattern and have a development
> build step that would convert npm packages into modules built to run in
> a new type of ResourceLoader that would probably be a hybrid of the
> current RL and npm. However you would need to take care to exclude
> things or else you'll end up with a script that will duplicate the
> source of a bunch of other modules. You also have to be wary of handling
> the cases where npm intentionally duplicates a library.
>
> This actually may be even more difficult than it sounds due to npm
> packages that require modules inside a package rather than just the main
> module. You might need to go even deeper into browserify, substituting
> one of the last steps so it outputs a different format with objects that
> have metadata on the individual modules intended for use in a special
> loader instead of as a plain script tag.
>
> Current RL on its own just won't work for this. Globals don't work well
> in the module pattern and so the ability to get a result from require()
> 

Re: [Wikitech-l] [RFC/Summit] `npm install mediawiki-express`

[C. Scott Ananian]

I view it as partly an effort to counteract the perceived complexity of
running a forest full of separate services.  It's fine to say they're all
preinstalled in this VM image, but that's still a lot of complexity to dig
through: where are the all the servers? What ports are they listening all?
Did one of them crash?  How do I restart it?

For some users, the VM (or an actual server farm) is indeed the right
solution.  But this was an attempt to see if I could recapture the
"everything's here in this one process (and one code tree)" simplicity for
those for whom that's good enough.
  --scott
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [RFC] Hygienic templates

[C. Scott Ananian]

On Fri, Oct 2, 2015 at 10:50 AM, Brad Jorsch (Anomie) <bjor...@wikimedia.org
> wrote:

> On Thu, Oct 1, 2015 at 6:43 PM, C. Scott Ananian <canan...@wikimedia.org>
> wrote:
> > Phabricator task for this RFC: https://phabricator.wikimedia.org/T114445
>
> FYI, hiding this at the end of the email tends to encourage email
> replies rather than replies in Phab.
>

That was intentional; the wording in
https://www.mediawiki.org/wiki/Requests_for_comment/Process suggested that
email conversation should be pursued orthogonally to discussion on the Phab
ticket.  The phab task links to the email discussion archives, so nothing's
going to be lost.

If you're in the habit of using phab for discussion, you didn't need to see
this email: phab would have notified you as soon as I created the task in
the RFC project. The mailing list thread is for people who would prefer to
discuss (or ask questions) outside phab.

Please correct me if I'm wrong; I'm just trying to follow the process
outlined in the wiki.
 --scott

-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l