Re: [Wikitech-l] Parsoid's progress

[MZMcBride]

C. Scott Ananian wrote:
>1) no plan survives first encounter with the enemy.  Parsoid was going to
>be simpler than the PHP parser, Parsoid was going to be written in PHP,
>then C, then prototyped in JS for a later implementation in C, etc.  It
>has varied over time as we learned more about the problem.  It is
>currently written in node.js and probably is at least the same order of
>complexity as the existing PHP parser.

Hrm.

>In many cases Parsoid could be greatly simplified if we didn't have to
>maintain compatibility with various strange corner cases in the PHP
>parser.

I guess this is the part that I'm still struggling with. If the PHP parser
is/was already doing the job of converting to wikitext to HTML, why would
that need to be rewritten in Node.js? Wouldn't it have been simpler to
make the HTML output more verbose in the PHP parser so that it could
cleanly round-trip? I'm still not clear where Node.js (or C or JavaScript)
came into this. I heard there were performance concerns with the PHP
parser. Was that the case?

I'm mostly just curious... you can't un-milk the cow, as they say.

>But note that even as a full parser replacement Parsoid depends on the PHP
>API in a large number of ways: imageinfo, siteinfo, language information,
>localized keywords for images, etc.  The idea of "independence" is
>somewhat vague.

Hrm.

MZMcBride



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Parsoid's progress

[MZMcBride]

Thank you both for the detailed replies. They were very helpful and I feel
like I have a better understanding now. I'm still trying to wrap my head
around Parsoid, its implementation, and how it fits in with the larger
future of MediaWiki development.

Subramanya Sastry wrote:
>The core parser has the following components:
>
>* preprocessing that expands transclusions, extensions (including
>  Scribunto), parser functions, include directives, etc. to wikitext
>* wikitext parser that converts wikitext to html
>* Tidy that runs on the html produced by wikitext parser and fixes up
>  malformed html
>
>Parsoid right now replaces the last two of the three components, but in
>a way that enables all of the functionality stated earlier.

Are you saying Parsoid can act as a replacement for HTMLTidy? That seems
like a pretty huge win. Replacing Tidy has been a longstanding goal:
.

>But, there are several directions this can go from here (including
>implementing a preprocessor in Parsoid, for example). However, note that
>this discussion is not entirely about Parsoid but also about shared
>hosting support, mediawiki packaging, pure PHP mediawiki install,
>HTML-only wikis, etc. All those other decisions inform what Parsoid
>should focus on and how it evolves.

I think this is very well put. There's definitely a lot to think about.

MZMcBride



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] wfRunHooks deprecation

[Gergo Tisza]

On Tue, Jan 20, 2015 at 4:21 PM, Chad  wrote:

> > I just noticed wfRunHooks got deprecated. The hook mechanism is heavily
> > depended on by extensions. So if it is going away, what will it be
> replaced
> > by? There is no hint of this in the method doc.
> >
> Hooks::run().
>

Does the new syntax offer any advantage over the old one?
Assuming that we want to switch to non-static function calls eventually
(which I hope is the case), wouldn't it be friendlier towards extension
maintainers to only deprecate once we are there, instead of forcing them to
update twice?
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] new extension

[Gergo Tisza]

On Mon, Jan 19, 2015 at 10:00 AM, Tyler Romeo  wrote:

> >  * Do you think the extension should use the extmetadata property of
> >ApiQueryImageInfo instead of a its own module?
> >  * Is it advisable to store validation data permanently in the database?
>
> I have no idea about this, but it does seem that the metadata is propagated
> to the oldimage table when a new one is uploaded, so it would fulfill your
> above question about storing old revisions' validation status.
>

metadata is data generated from the file. It has built-in storage and
invalidation mechanisms that are based on file upload / purge.
extmetadata is assumed to come from some other source, and providers need
to handle invalidation (and permanent storage, if desirable) manually.
Thus, metadata would be a better fit in theory, but I don't think it offers
any mechanism currently for extensions to hook into it. So I don't think
you are any worse off by writing a self-contained extension.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] wfRunHooks deprecation

[Jeroen De Dauw]

Hey,

> Hooks::run()

Oh, in that case there is not much to worry about.

Added a note in
https://gerrit.wikimedia.org/r/#/c/186115/1/includes/GlobalFunctions.php

Cheers

--
Jeroen De Dauw - http://www.bn2vs.com
Software craftsmanship advocate
Evil software architect at Wikimedia Germany
~=[,,_,,]:3
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] wfRunHooks deprecation

[Daniel Friesen]

On 2015-01-20 4:21 PM, Chad wrote:
> On Tue Jan 20 2015 at 4:20:47 PM Jeroen De Dauw 
> wrote:
>
>> Hey,
>>
>> I just noticed wfRunHooks got deprecated. The hook mechanism is heavily
>> depended on by extensions. So if it is going away, what will it be replaced
>> by? There is no hint of this in the method doc.
>>
>>
> Hooks::run().
>
> -Chad
Heh, yay.

To bad we're not deprecating $wgHooks -> Hooks::register.
I've had ideas that are impossible to do with $wgHooks.

((IIRC it was something about storing a light stack in debug mode of
where Hooks::register is called so hook related errors, deprecation
notices, etc... (or it might have been for a different kind of error)
can actually tell you where the hook was registered))

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] wfRunHooks deprecation

[Chad]

On Tue Jan 20 2015 at 4:20:47 PM Jeroen De Dauw 
wrote:

> Hey,
>
> I just noticed wfRunHooks got deprecated. The hook mechanism is heavily
> depended on by extensions. So if it is going away, what will it be replaced
> by? There is no hint of this in the method doc.
>
>
Hooks::run().

-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] wfRunHooks deprecation

[Jeroen De Dauw]

Hey,

I just noticed wfRunHooks got deprecated. The hook mechanism is heavily
depended on by extensions. So if it is going away, what will it be replaced
by? There is no hint of this in the method doc.

Cheers

--
Jeroen De Dauw - http://www.bn2vs.com
Software craftsmanship advocate
Evil software architect at Wikimedia Germany
~=[,,_,,]:3
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The future of shared hosting

[James Douglas]

> But there's plenty of room for other initiatives (some could even make
money out of this :)).

For what it's worth, this brings to mind a couple interesting examples of
this pattern:
* https://travis-ci.com/ (and its free counterpart, https://travis-ci.org/),
the hosted version of https://github.com/travis-ci/travis-ci
* https://gitlab.com/, the hosted version of
https://github.com/gitlabhq/gitlabhq


On Tue, Jan 20, 2015 at 3:37 PM, Markus Glaser  wrote:

> Hello everyone,
>
> On 19/01/15 06:47, Tim Starling wrote:
> > As long as there are no actual reasons for dropping pure-PHP core
> functionality,
> > the idea of WMF versus shared hosting is a false dichotomy.
> I kind of agree. Instead of seeing things in black and white, aka shared
> hosting or not, we should take a look at the needs of users who run their
> MW on a shared hosting. What exactly do they consider "core functionality"?
> I don't think we actually know the answer yet. To me, it seems very likely
> that MWs on shared hosts are a starting base into the MW world. Probably,
> their admins are mostly not technologically experienced. In the near
> future, most of them will want to see VE on their instances for improved
> user experience. But do they really care about wikicode? Or do they care
> about some functionality that solves their problems. I could imagine,
> templating is one of the main reasons to ask for wikicode. Can we, then,
> support templating in pure HTML versions of parsoid? Are there other
> demands and solutions? What I mean is: there are many shades of [any color
> you like], in order to make users outside the WMF happy.
>
> I see shared hosts somewhere in a middle position in terms of skills
> needed to run and in terms of dedication to the site. On the "lower"
> ground, there are test instances run on local computers. These can be
> supported, for example, with vagrant setups, in order to make it very easy
> to get started. On the "upper" level, there are instances that run on
> servers with root access, vms, in clouds, etc. They can be supported, for
> instance, with modular setup instructions, packages, predefined machines,
> puppet and other install scripts in order to get a proper setup. So shared
> hosting is a special case, then, but it seems to have a significant base of
> users and supporters.
>
> While the current SOA approach makes things more complex in terms of
> technologies one needs to support in order to run a setup that matches one
> of the top 5 websites, it also makes things easier in terms of modularity,
> if we do it right. So, for example, we (tm) could provide a lightweight PHP
> implementation of parsoid. Or someone (tm) runs a parsoid service somewhere
> in the net.
>
> The question is, then, who should be "someone". Naturally, WMF seems to be
> predestined to lay the ground, e.g. by publishing setup instructions,
> interfaces, puppet rules, etc. But there's plenty of room for other
> initiatives (some could even make money out of this :)). An ecosystem
> around MediaWiki can help do the trick. But here's the crucial bit: We will
> only get a healthy ecosystem around MediaWiki, if things are reliable in a
> way. If the developer community and the WMF commits to support such an
> environment. In the current situation, there's so much insecurity I doubt
> anyone will seriously consider putting a lot of effort in, say, a PHP
> parsoid port (I'd be happy if someone proves me wrong).
>
> So, to make a long story short: Let's look forward and try to find
> solutions. MediaWiki is an amazing piece of software and we should never
> stop to salutate and support the hundreds of thousands of people that are
> using it as a basis of furthering the cause of free knowledge.
>
> Best,
> Markus
>
> -Ursprüngliche Nachricht-
> Von: wikitech-l-boun...@lists.wikimedia.org [mailto:
> wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Tim Starling
> Gesendet: Montag, 19. Januar 2015 06:47
> An: wikitech-l@lists.wikimedia.org
> Betreff: Re: [Wikitech-l] The future of shared hosting
>
> On 16/01/15 17:38, Bryan Davis wrote:
> > The solution to these issues proposed in the RFC is to create
> > independent services (eg Parsoid, RESTBase) to implement features that
> > were previously handled by the core MediaWiki application. Thus far
> > Parsoid is only required if a wiki wants to use VisualEditor. There
> > has been discussion however of it being required in some future
> > version of MediaWiki where HTML is the canonical representation of
> > articles {{citation needed}}.
>
> Parsoid depends on the MediaWiki parser, it calls it via api.php. It's not
> a complete, standalone implementation of wikitext to HTML transformation.
>
> HTML storage would be a pretty simple feature, and would allow third-party
> users to use VE without Parsoid. It's not so simple to use Parsoid without
> the MediaWiki parser, especially if you want to support all existing
> extensions.
>
> So, as currently proposed, HTML storage is

Re: [Wikitech-l] The future of shared hosting

[Markus Glaser]

Hello everyone,

On 19/01/15 06:47, Tim Starling wrote:
> As long as there are no actual reasons for dropping pure-PHP core 
> functionality, 
> the idea of WMF versus shared hosting is a false dichotomy.
I kind of agree. Instead of seeing things in black and white, aka shared 
hosting or not, we should take a look at the needs of users who run their MW on 
a shared hosting. What exactly do they consider "core functionality"? I don't 
think we actually know the answer yet. To me, it seems very likely that MWs on 
shared hosts are a starting base into the MW world. Probably, their admins are 
mostly not technologically experienced. In the near future, most of them will 
want to see VE on their instances for improved user experience. But do they 
really care about wikicode? Or do they care about some functionality that 
solves their problems. I could imagine, templating is one of the main reasons 
to ask for wikicode. Can we, then,  support templating in pure HTML versions of 
parsoid? Are there other demands and solutions? What I mean is: there are many 
shades of [any color you like], in order to make users outside the WMF happy.

I see shared hosts somewhere in a middle position in terms of skills needed to 
run and in terms of dedication to the site. On the "lower" ground, there are 
test instances run on local computers. These can be supported, for example, 
with vagrant setups, in order to make it very easy to get started. On the 
"upper" level, there are instances that run on servers with root access, vms, 
in clouds, etc. They can be supported, for instance, with modular setup 
instructions, packages, predefined machines, puppet and other install scripts 
in order to get a proper setup. So shared hosting is a special case, then, but 
it seems to have a significant base of users and supporters.

While the current SOA approach makes things more complex in terms of 
technologies one needs to support in order to run a setup that matches one of 
the top 5 websites, it also makes things easier in terms of modularity, if we 
do it right. So, for example, we (tm) could provide a lightweight PHP 
implementation of parsoid. Or someone (tm) runs a parsoid service somewhere in 
the net. 

The question is, then, who should be "someone". Naturally, WMF seems to be 
predestined to lay the ground, e.g. by publishing setup instructions, 
interfaces, puppet rules, etc. But there's plenty of room for other initiatives 
(some could even make money out of this :)). An ecosystem around MediaWiki can 
help do the trick. But here's the crucial bit: We will only get a healthy 
ecosystem around MediaWiki, if things are reliable in a way. If the developer 
community and the WMF commits to support such an environment. In the current 
situation, there's so much insecurity I doubt anyone will seriously consider 
putting a lot of effort in, say, a PHP parsoid port (I'd be happy if someone 
proves me wrong).  

So, to make a long story short: Let's look forward and try to find solutions. 
MediaWiki is an amazing piece of software and we should never stop to salutate 
and support the hundreds of thousands of people that are using it as a basis of 
furthering the cause of free knowledge.

Best,
Markus

-Ursprüngliche Nachricht-
Von: wikitech-l-boun...@lists.wikimedia.org 
[mailto:wikitech-l-boun...@lists.wikimedia.org] Im Auftrag von Tim Starling
Gesendet: Montag, 19. Januar 2015 06:47
An: wikitech-l@lists.wikimedia.org
Betreff: Re: [Wikitech-l] The future of shared hosting

On 16/01/15 17:38, Bryan Davis wrote:
> The solution to these issues proposed in the RFC is to create 
> independent services (eg Parsoid, RESTBase) to implement features that 
> were previously handled by the core MediaWiki application. Thus far 
> Parsoid is only required if a wiki wants to use VisualEditor. There 
> has been discussion however of it being required in some future 
> version of MediaWiki where HTML is the canonical representation of 
> articles {{citation needed}}.

Parsoid depends on the MediaWiki parser, it calls it via api.php. It's not a 
complete, standalone implementation of wikitext to HTML transformation.

HTML storage would be a pretty simple feature, and would allow third-party 
users to use VE without Parsoid. It's not so simple to use Parsoid without the 
MediaWiki parser, especially if you want to support all existing extensions.

So, as currently proposed, HTML storage is actually a way to reduce the 
dependency on services for non-WMF wikis, not to increase it.

Based on recent comments from Gabriel and Subbu, my understanding is that there 
are no plans to drop the MediaWiki parser at the moment.

> This particular future may or may not be far off on the calendar, but 
> there are other services that have been proposed (storage service, 
> REST content API) that are likely to appear in production use at least 
> for the Foundation projects within the next year.

There is a proposal to move revision storage to Cassandra, p

Re: [Wikitech-l] Announcement: Chad Horohoe joins the Release Engineering team

[James Forrester]

On 20 January 2015 at 14:39, Greg Grossmeier  wrote:

> I'm entirely pleased to announce that Chad Horohoe is joining the
> Wikimedia Foundation's Release Engineering Team.
>

​Congratulations, ​Release Engineering Team and Chad. Great news! Really
looking forward to working together.

J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforres...@wikimedia.org | @jdforrester
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Announcement: Chad Horohoe joins the Release Engineering team

[Greg Grossmeier]

I'm entirely pleased to announce that Chad Horohoe is joining the
Wikimedia Foundation's Release Engineering Team.

This community needs no introduction to Chad so I thought I’d share a
little bit of my personal experience with him.

I first worked with Chad on the MediaWiki Core Team; in fact, Chad was
one of the people who interviewed me for my initial position at WMF.
That's when I got to see first hand Chad's relentlessness and strong
(and vocal) commitment to the mission of the WMF. He tells you when you
think you’re wrong (in a nice way) but most importantly helps you make
things right. In short, he follows up his critique with patches (both
literally and figuratively).

Chad will be an obvious asset to the Release Engineering team; his
experience with Gerrit, his (already deep) knowledge of Phabricator, his
understanding of how all the pieces of our system work together (which
will help Beta Cluster), and his familiarity (and annoyances) of our
deployment tooling.

Even though I can’t express through email[0] my excitement that Chad is
joining the Release Engineering team please join me in welcoming him to
his new role.


Greg


[0] !!1!!!111eleven

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Brian Wolff]

On Jan 20, 2015 12:34 PM, "Brion Vibber"  wrote:
>
> Quick update:
>
> I've had a great experience working on our mobile apps, but it's time to
> get back to core MediaWiki and help "clean my own house"... now that we've
> got Mobile Apps fully staffed I'm leaving the mobile department and will
be
> reporting directly to Damon within WMF Engineering.
>
> First -- huge thanks to Monte and Dan and Kristen and Dmitry and Bernd and
> of course Tomasz!! and everybody else who's been awesome in Mobile Apps --
> and also to the rest of the mobile team, who have become too many to list
> in a concise email. :)
>
>
> For the moment I'm going to get back up to speed with the Architecture
> Committee and push at general MediaWiki issues. As we determine the fate
of
> committees and narrow down what are our priority projects, my focus may
> narrow a to getting some particular things done over the next months.
>
>
> A few general notes:
>
> * Working in mobile apps reminded me how important our APIs are -- our
> ability to make flexible interfaces that work in different form factors
and
> different technology stacks is dependent on maintaining a good API. This
> needs work. :)
>
> This doesn't just mean interacting with "api.php" -- we need clean
> configuration, data, etc interfaces as well, especially if we want people
> to contribute in ways other than raw text editing. There's a lot to clean
> up...
>
> * Mobile mobile mobile! I've heard some folks complain that while there's
a
> lot of talk about "mobile-first" and similar there aren't always concrete
> explanations yet of what that means. I hope to bring some of the
excitement
> we've seen in Mobile about Wikidata, better queries, better visual/user
> interaction design, and generally making things *work for users*.
>
> * Breaking or working around the "PHP barrier" for third-party MediaWiki
> users: I hope to get the question of services resolved one way or another
> -- either by us officially dropping "shared PHP hosting" support or by
> making sure we have "pure PHP" implementations of things that are required
> to operate -- which is mostly dependent on having good interfaces and APIs
> so that multiple implementations can be written and maintained
compatibly...
>
> * Future stuff -- new media types, narrow-field editing, natural language
> queries? WHO KNOWS! I'll be researching more crazy stuff in my additional
> time.
>
>
> I'll see many of you at the Dev Summit soon enough -- don't be shy about
> pestering me with concerns and ideas about priorities. :)
>
> -- brion
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Exciting times! Im sure this will result in many new great things for
MediaWiki.

--bawolff
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[Helder .]

On Tue, Jan 20, 2015 at 7:53 PM, Brian Wolff  wrote:
> My second reason for being skeptical is im mostly unclear on what the
> benefits are over wikitext storage (this is the first time ive heard of the
> ve w/o parsoid thing. Are there other benefits? Simplifying parser cache by
> not having parser cache?)
I don't know if this helps but HTML-only wikis are on the roadmap at
least since May 2014:
https://www.mediawiki.org/w/index.php?title=Parsoid/Roadmap/2014_15#HTML-only_wiki_support

Helder

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[Brian Wolff]

On Jan 20, 2015 5:53 PM, "Brian Wolff"  wrote:
>
>
> On Jan 20, 2015 4:22 PM, "James Forrester" 
wrote:
> >
> > On 20 January 2015 at 12:04, Jeroen De Dauw 
wrote:
> >
> > > ​>​
> > >- ​Get rid of wikitext on the server-side.
> > > >   - HTML storage only. Remove MWParser from the codebase. All
> > > >   extensions that hook into wikitext (so, almost all of them?)
will
> > > > need to
> > > >   be re-written.
> > > >
> > >
> > > Just to confirm: this is not actually on the WMF roadmap right? :)
> > >
> >
> > ​It's certainly not what I'm working on for the next year or so. It is
> > unlikely to be something we do for WMF usage; it's more valuable to
people
> > that want VisualEditor but want PHP-only (or don't want Node) for the
> > server.
> >
> > J.
> > --
>
> Hypothetically how would this work? Wouldnt you still need parsoid to
verify the html corresponds to some legit wikitext? Otherwise how would you
know its safe?
>
> Since we are somewhat having a discussion about this (i recognize that
this isnt a "real" discussion in the sense that there isnt a full technical
proposal, or any concrete plans to actually do it in near future, just a
wild idea that some people like), one of the reasons im somewhat skeptical
about this idea, is if there is an xss issue, you seem much more screwed
with html storage, since now the bad stuff is in the canonical
representation, instead of wikitext storage where you can just fix the
parser, potentially purge parser cache, and then you are 100% certain your
wiki is clean.
>
> My second reason for being skeptical is im mostly unclear on what the
benefits are over wikitext storage (this is the first time ive heard of the
ve w/o parsoid thing. Are there other benefits? Simplifying parser cache by
not having parser cache?)
>
> I may be misinterpreting how such a thing is proposed to work. Im not
very familar with parsoid and associated things.
>
> --bawolff

And the other thread which i hadn't read at the time of writing this
answers my question in that html verification is something yet to be
determined :)

--bawolff
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[Brian Wolff]

On Jan 20, 2015 4:22 PM, "James Forrester"  wrote:
>
> On 20 January 2015 at 12:04, Jeroen De Dauw 
wrote:
>
> > ​>​
> >- ​Get rid of wikitext on the server-side.
> > >   - HTML storage only. Remove MWParser from the codebase. All
> > >   extensions that hook into wikitext (so, almost all of them?)
will
> > > need to
> > >   be re-written.
> > >
> >
> > Just to confirm: this is not actually on the WMF roadmap right? :)
> >
>
> ​It's certainly not what I'm working on for the next year or so. It is
> unlikely to be something we do for WMF usage; it's more valuable to people
> that want VisualEditor but want PHP-only (or don't want Node) for the
> server.
>
> J.
> --

Hypothetically how would this work? Wouldnt you still need parsoid to
verify the html corresponds to some legit wikitext? Otherwise how would you
know its safe?

Since we are somewhat having a discussion about this (i recognize that this
isnt a "real" discussion in the sense that there isnt a full technical
proposal, or any concrete plans to actually do it in near future, just a
wild idea that some people like), one of the reasons im somewhat skeptical
about this idea, is if there is an xss issue, you seem much more screwed
with html storage, since now the bad stuff is in the canonical
representation, instead of wikitext storage where you can just fix the
parser, potentially purge parser cache, and then you are 100% certain your
wiki is clean.

My second reason for being skeptical is im mostly unclear on what the
benefits are over wikitext storage (this is the first time ive heard of the
ve w/o parsoid thing. Are there other benefits? Simplifying parser cache by
not having parser cache?)

I may be misinterpreting how such a thing is proposed to work. Im not very
familar with parsoid and associated things.

--bawolff
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[James HK]

Hi,

On 1/21/15, Daniel Friesen  wrote:
> On 2015-01-20 12:21 PM, James HK wrote:
>> Hi,
>>
>>>- ​Get rid of wikitext on the server-side.
>>>   - HTML storage only. Remove MWParser from the codebase. All
>>>   extensions that hook into wikitext (so, almost all of them?) will
>>> need
>>> to
>>>   be re-written.
>> You gotta to be kidding. It seems we are a bit out of sync here with
>> the real world for proposing (or demanding not sure if above is being
>> seriously considered) something like "All extensions that hook into
>> wikitex ... will need to be re-written.".
> Parsoid, etc... can already handle parser functions and tag hooks.
> So I'm not certain but, he may be talking specifically about extensions
> that hook into WikiText to implement their own custom syntax like
> [[Foo::Bar]].

We certainty want to avoid a situation where we have to write
spaghetti code just because of

if ( $parser === 'Parsoid' ) {
return $this->doSomethingToBeAbleToHandle( '[[Foo::Bar]]' );
}

Cheers

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[Daniel Friesen]

On 2015-01-20 12:21 PM, James HK wrote:
> Hi,
>
>>- ​Get rid of wikitext on the server-side.
>>   - HTML storage only. Remove MWParser from the codebase. All
>>   extensions that hook into wikitext (so, almost all of them?) will need
>> to
>>   be re-written.
> You gotta to be kidding. It seems we are a bit out of sync here with
> the real world for proposing (or demanding not sure if above is being
> seriously considered) something like "All extensions that hook into
> wikitex ... will need to be re-written.".
Parsoid, etc... can already handle parser functions and tag hooks.
So I'm not certain but, he may be talking specifically about extensions
that hook into WikiText to implement their own custom syntax like
[[Foo::Bar]].

> Just because it seems favourable to the WMF cluster to run Parsoid services 
> it doesn't mean
> anyone else needs a bunch of extra node.js servers to handle Parsoid
> just to store and manage some text in mediawiki.
As explained in the other topic, VisualEditor can edit the DOM without
Parsoid, so this doesn't mean everyone would have to have Parsoid servers.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[S Page]

On Tue, Jan 20, 2015 at 9:22 AM, Tei  wrote:

> a open door to things that can freak us again.
>
^ world's greatest job title for Brion's business card

Robla wrote

> Brion,let me know how I can help!

Ditto.

Great news indeed.
-- 
=S Page  WMF Tech writer
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Congratulations to MediaWiki Farmers User Group for being approved as a Wikimedia User Group

[Legoktm]

On 01/17/2015 03:54 PM, Gregory Varnum wrote:
> Greetings,
> 
> Please join the Affiliations Committee in congratulating the MediaWiki
> Farmers User Group on their official approval as a Wikimedia User Group:
> https://meta.wikimedia.org/wiki/Affiliations_Committee/Resolutions/MediaWiki_Farmers_User_Group_-_Liaison_approval,_January_2015

Thank you for all your help in the process!

> The MediaWiki Farmers User Group is "A user group of third-party developers
> who work on wiki farms. Our mission is to improve and standardize the way
> MediaWiki wiki farms are setup and run."
> 
> Anyone interested in more information about the group can visit:
> https://www.mediawiki.org/wiki/Project:MediaWiki_Farmers_user_group

We will be having our first meeting[1] at the developer summit, and will
post meeting minutes on-wiki for those who aren't able to attend.
We'll also have an IRC meeting sometime after the summit, specifics
still to be determined.

[1] https://phabricator.wikimedia.org/T87240

-- Legoktm

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] VE without Parsoid

[Stas Malyshev]

Hi!

> 2.  HTML validation - our current security model relies on the HTML
> being generated server-side by a wikitext parser.  If we cut wikitext
> out of the loop, we'll need some other way of ensuring that people
> can't inject arbitrary Javascript/Flash embedding/Java
> applet/ActionScript/iframe or any other security horrors.

There are tools like HTML Purifier which are pretty good at it, though
performance of those are not stellar, especially on big texts. The
Purifier pretty much disassembles it into DOM, validates that, throws
out what it doesn't like and reassembles it back. Which is not very fast
in PHP, but is pretty strict. Still, there's a chance people could sneak
something past it.

-- 
Stas Malyshev
smalys...@wikimedia.org

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[James Forrester]

On 20 January 2015 at 12:04, Jeroen De Dauw  wrote:

> ​>​
>- ​Get rid of wikitext on the server-side.
> >   - HTML storage only. Remove MWParser from the codebase. All
> >   extensions that hook into wikitext (so, almost all of them?) will
> > need to
> >   be re-written.
> >
>
> Just to confirm: this is not actually on the WMF roadmap right? :)
>

​It's certainly not what I'm working on for the next year or so. It is
unlikely to be something we do for WMF usage; it's more valuable to people
that want VisualEditor but want PHP-only (or don't want Node) for the
server.

J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforres...@wikimedia.org | @jdforrester
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[James HK]

Hi,

>- ​Get rid of wikitext on the server-side.
>   - HTML storage only. Remove MWParser from the codebase. All
>   extensions that hook into wikitext (so, almost all of them?) will need
> to
>   be re-written.

You gotta to be kidding. It seems we are a bit out of sync here with
the real world for proposing (or demanding not sure if above is being
seriously considered) something like "All extensions that hook into
wikitex ... will need to be re-written.". Just because it seems
favourable to the WMF cluster to run Parsoid services it doesn't mean
anyone else needs a bunch of extra node.js servers to handle Parsoid
just to store and manage some text in mediawiki.

Cheers

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 2.0 (was: No more Architecture Committee?)

[Jeroen De Dauw]

Hey,

FYI: we had this discussion for SMW some time ago. We where at version 1.9,
following a naming pattern very similar to the MediaWiki one. The question
there was if the next release containing a break should be either 1.10 (no
change), 2.0 (following semver) or 10.0 (dropping the "1."). People
generally preferred 2.0, mostly because "10.0 seems like such a big jump".
So this is what we went with. And while the 2.0 release was not more
significant than 1.9 or 1.8, no one seems to have gotten seriously confused
about it.

   - ​Get rid of wikitext on the server-side.
>   - HTML storage only. Remove MWParser from the codebase. All
>   extensions that hook into wikitext (so, almost all of them?) will
> need to
>   be re-written.
>

Just to confirm: this is not actually on the WMF roadmap right? :)

Cheers

--
Jeroen De Dauw - http://www.bn2vs.com
Software craftsmanship advocate
Evil software architect at Wikimedia Germany
~=[,,_,,]:3
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Getting the full URL of an image

[Daniel Friesen]

Also try using Title::makeTitleSafe instead of Title:makeTitle.

Your $fileName might not be in the proper dbKey format (use of spaces,
etc...) in which case the Title you get would be corrupt since makeTitle
doesn't sanitize the text you give.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2015-01-20 11:46 AM, Brion Vibber wrote:
> I think you have a double 'File:' prefix in there which may be confusing
> things. Try Title::newFromText($fileName) if $fileName already includes
> 'File:' prefix.
>
> -- brion
>
> On Tue, Jan 20, 2015 at 11:42 AM, Jeroen De Dauw 
> wrote:
>
>> Hey,
>>
>> Now trying this code: wfFindFile( $title );
>>
>> With this title object as input: http://pastebin.com/7B9n1NyN (constructed
>> with Title::makeTitle( NS_FILE, $fileName ))
>>
>> The output is still false. At the same time, the page definitely exists,
>> and if I put "[[File:Blue marker.png]]" in a wiki page, it shows up just
>> fine. I double checked my file permissions, and they seem fine. What am I
>> missing?
>>
>> Cheers
>>
>> --
>> Jeroen De Dauw - http://www.bn2vs.com
>> Software craftsmanship advocate
>> Evil software architect at Wikimedia Germany
>> ~=[,,_,,]:3
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Parsoid's progress

[Subramanya Sastry]

Some quick comments.

As has already been alluded to, Parsoid does a couple different things.

* It converts wikitext to html (in such a way that edits to the html
  can be serialized back to wikitext without introducing dirty diffs
  in the wikitext).
* It converts html to wikitext (in such a way that edits to the wikitext
  preserves html semantics). There are caveats here in that Parsoid
  doesn't yet handle arbitrary html that you might throw at it, but 
insofaras

  the HTML conforms to the DOM spec [1], Parsoid should do a good job of
  serializing it to wikitext.

This bidirectionality means that Parsoid can support clients that don't
need to deal with wikitext directly knowing that Parsoid can go both ways.
Amir has mentioned Content Translation. See the full list of clients 
here [3].


This support for bidirectional conversion between wikitext and HTML is
non-trivial. See "Lossless conversion" section and other details in [2]. 
Getting

to this stage where Parsoid is in terms of rendering and bidrectionality has
required us to work through a lot of issues and edge cases given that 
editing
requires HTML semantics and wikitext and transclusions is string-based. 
Parsoid
can map a DOM node to a substring of wikitext that generated it, and 
that is also
a non-trivial achievement. See the tech talk here [4]. I'm skipping the 
details of
the different levels of testing that we implement to achieve this, but 
that has been
a substantial part of getting to this part and being able to deploy 
seamlessly on a

regular basis [5] largely without incident.

As for the other part about preprocessing, yes, Parsoid currently relies on
the Mediawiki API.

The core parser has the following components:

* preprocessing that expands transclusions, extensions (including 
Scribunto),

  parser functions, include directives, etc. to wikitext
* wikitext parser that converts wikitext to html
* Tidy that runs on the html produced by wikitext parser and fixes up
  malformed html

Parsoid right now replaces the last two of the three components, but in
a way that enables all of the functionality stated earlier. I'll skip 
the historical and
technical reasons right now why we haven't put energy and resources into 
this

component of Parsoid, but in brief, we found it more important to enable the
bidirectional functionality and supporting clients and reuse the 
preprocessing

functionality via the mediawiki API.

But, there are several directions this can go from here (including 
implementing

a preprocessor in Parsoid, for example). However, note that this discussion
is not entirely about Parsoid but also about shared hosting support,
mediawiki packaging, pure PHP mediawiki install, HTML-only wikis, etc.
All those other decisions inform what Parsoid should focus on and how it 
evolves.


Subbu.

[1] http://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec
[2] 
https://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with-the-web/

[3] http://www.mediawiki.org/wiki/Parsoid/Users
[4] https://www.youtube.com/watch?v=Eb5Ri0xqEzk with slides @ 
https://commons.wikimedia.org/wiki/File:Parsoid.techtalk.apr15.2014.pdf

[5] https://www.mediawiki.org/wiki/Parsoid/Deployments

On 01/20/2015 08:02 AM, C. Scott Ananian wrote:

I believe Subbu will follow up with a more complete response, but I'll note
that:

1) no plan survives first encounter with the enemy.  Parsoid was going to
be simpler than the PHP parser, Parsoid was going to be written in PHP,
then C, then prototyped in JS for a later implementation in C, etc.  It has
varied over time as we learned more about the problem.  It is currently
written in node.js and probably is at least the same order of complexity as
the existing PHP parser.  It is, however, built on slightly more solid
foundations, so its behavior is more regular than the PHP parser in many
places -- although I've been submitting patches to the core parser where
necessary to try to bring them closer together.  (c.f.
https://gerrit.wikimedia.org/r/180982 for the most recent of these.)  And,
of course, Parsoid emits well-formed HTML which can be round-tripped.

In many cases Parsoid could be greatly simplified if we didn't have to
maintain compatibility with various strange corner cases in the PHP parser.

2) Parsoid contains a partial implementation of the PHP expandtemplates
module.  It was decided (I think wisely) that we didn't really gain
anything by trying to reimplement this on the Parsoid side, though, and it
was better to use the existing PHP code via api.php.  The alternative would
be to basically reimplement quite a lot of mediawiki (lua embedding, the
various parser functions extensions, etc) in node.js.  This *could* be done
-- there is no technical reason why it cannot -- but nobody thinks it's a
good idea to spend time on right now.

But the expandtemplates stuff basically works.   As I said, it doesn't
contain all the crazy extensions that we use on the main WMF sites, but it
would be re

Re: [Wikitech-l] Getting the full URL of an image

[Brion Vibber]

I think you have a double 'File:' prefix in there which may be confusing
things. Try Title::newFromText($fileName) if $fileName already includes
'File:' prefix.

-- brion

On Tue, Jan 20, 2015 at 11:42 AM, Jeroen De Dauw 
wrote:

> Hey,
>
> Now trying this code: wfFindFile( $title );
>
> With this title object as input: http://pastebin.com/7B9n1NyN (constructed
> with Title::makeTitle( NS_FILE, $fileName ))
>
> The output is still false. At the same time, the page definitely exists,
> and if I put "[[File:Blue marker.png]]" in a wiki page, it shows up just
> fine. I double checked my file permissions, and they seem fine. What am I
> missing?
>
> Cheers
>
> --
> Jeroen De Dauw - http://www.bn2vs.com
> Software craftsmanship advocate
> Evil software architect at Wikimedia Germany
> ~=[,,_,,]:3
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Getting the full URL of an image

[Jeroen De Dauw]

Hey,

Now trying this code: wfFindFile( $title );

With this title object as input: http://pastebin.com/7B9n1NyN (constructed
with Title::makeTitle( NS_FILE, $fileName ))

The output is still false. At the same time, the page definitely exists,
and if I put "[[File:Blue marker.png]]" in a wiki page, it shows up just
fine. I double checked my file permissions, and they seem fine. What am I
missing?

Cheers

--
Jeroen De Dauw - http://www.bn2vs.com
Software craftsmanship advocate
Evil software architect at Wikimedia Germany
~=[,,_,,]:3
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Antoine Musso]

Le 20/01/2015 08:34, Brion Vibber a écrit :
> I've had a great experience working on our mobile apps, but it's time to
> get back to core MediaWiki and help "clean my own house"... now that we've
> got Mobile Apps fully staffed I'm leaving the mobile department and will be
> reporting directly to Damon within WMF Engineering.

Welcome "back" Brion, you were very missed :-]

-- 
Antoine "hashar" Musso


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] VE without Parsoid

[James Forrester]

On 20 January 2015 at 11:08, Rob Lanphier  wrote:

> On Tue, Jan 20, 2015 at 10:54 AM, Mark A. Hershberger 
> wrote:
> > Tim Starling  writes:
> >> HTML storage would be a pretty simple feature, and would allow
> >> third-party users to use VE without Parsoid.
> >
> > I've been thinking about how to implement an alternative to Parsoid
> > so that users of MW could use VE without standing up a node.js server.
> >
> > This gives me hope.
> >
> > Could anyone elaborate on what VE needs from MW in order to operate
> > without Parsoid?  Maybe there is an architecture diagram I've been
> > missing?
>
> Funny, we were just talking about that in the office  :-)
>
> There are two things that are needed:
> 1.  ContentHandler support for storing HTML.  As Tim points out, this
> should be reasonably straightforward.  I imagine there is a long tail
> of small implementation gaps, but a prototype-quality implementation
> of this would probably be easy.
> 2.  HTML validation - our current security model relies on the HTML
> being generated server-side by a wikitext parser.  If we cut wikitext
> out of the loop, we'll need some other way of ensuring that people
> can't inject arbitrary Javascript/Flash embedding/Java
> applet/ActionScript/iframe or any other security horrors.
>
> #2 is a lot harder, and something we need to decide how we want to
> implement (e.g. is this something we do in PHP or Node?).
>

There's also #3 – doing all the dozens of things that the wikitext parser
does on ingestion. All the redlinks and categories tables, ​building a
user-land (not UI) HTML template system for transclusions, media updates,
*etc.* Not a trivial task.

​J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforres...@wikimedia.org | @jdforrester
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] VE without Parsoid

[Rob Lanphier]

On Tue, Jan 20, 2015 at 10:54 AM, Mark A. Hershberger  
wrote:
> Tim Starling  writes:
>> HTML storage would be a pretty simple feature, and would allow
>> third-party users to use VE without Parsoid.
>
> I've been thinking about how to implement an alternative to Parsoid
> so that users of MW could use VE without standing up a node.js server.
>
> This gives me hope.
>
> Could anyone elaborate on what VE needs from MW in order to operate
> without Parsoid?  Maybe there is an architecture diagram I've been
> missing?

Funny, we were just talking about that in the office  :-)

There are two things that are needed:
1.  ContentHandler support for storing HTML.  As Tim points out, this
should be reasonably straightforward.  I imagine there is a long tail
of small implementation gaps, but a prototype-quality implementation
of this would probably be easy.
2.  HTML validation - our current security model relies on the HTML
being generated server-side by a wikitext parser.  If we cut wikitext
out of the loop, we'll need some other way of ensuring that people
can't inject arbitrary Javascript/Flash embedding/Java
applet/ActionScript/iframe or any other security horrors.

#2 is a lot harder, and something we need to decide how we want to
implement (e.g. is this something we do in PHP or Node?).

Rob

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] VE without Parsoid

[Mark A. Hershberger]

Tim Starling  writes:

> HTML storage would be a pretty simple feature, and would allow
> third-party users to use VE without Parsoid.

I've been thinking about how to implement an alternative to Parsoid
so that users of MW could use VE without standing up a node.js server.

This gives me hope.

Could anyone elaborate on what VE needs from MW in order to operate
without Parsoid?  Maybe there is an architecture diagram I've been
missing?

Mark.

-- 
Mark A. Hershberger
NicheWork LLC
717-271-1084

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The future of shared hosting

[Mark A. Hershberger]

MZMcBride  writes:

> We should be proud of how popular MediaWiki has become across the
> Internet and we should support its growth because it ultimately
> supports our own. The more people engaged with and using MediaWiki,
> the better off we are.

Thank you for this.  I agree whole heartedly.  I'd like to see what we
can do to increase its usage.

Mark.

-- 
Mark A. Hershberger
NicheWork LLC
717-271-1084

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] I haven't looked lately...

[Federico Leva (Nemo)]

Really, James? WMF still has to cleanup the conversion from UseModWiki: 
https://phabricator.wikimedia.org/T2323
When bug 323 and others are fixed, I may believe you about 
billions-revisions conversions.


Nemo

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Rob Lanphier]

On Tue, Jan 20, 2015 at 10:06 AM, Greg Grossmeier  wrote:
> Great news Brion!
>
> I'm excited about what your energy will bring (back) to this part of the
> software stack.
>
> And I'll remain reserved and not make fanboy comments; professionalism
> and all.

What he said ^

My getting involved in MediaWiki development was heavily influenced by
watching Brion's friendly and clear direction of the project.  Brion,
let me know how I can help!

Rob

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki release management moves back to the WMF

[Bartosz Dziewoński]

Nice! It's going to be difficult for the new-old team to match Mark&Markus  
when it comes to release quality. ;)


--
Bartosz Dziewoński

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Parsoid's progress

[Arcane 21]

Given what I've seen so far, it might be best to aim for a gradual 
reimplementation of Parsoid features to make most of them work without a need 
for Parsoid, with the eventual goal of severing the need for Parsoid completely 
if possible. At any rate, the less the parser has to outsource, the less 
complicated things will be, correct?

> Date: Tue, 20 Jan 2015 11:02:10 -0500
> From: canan...@wikimedia.org
> To: wikitech-l@lists.wikimedia.org
> Subject: Re: [Wikitech-l] Parsoid's progress
> 
> I believe Subbu will follow up with a more complete response, but I'll note
> that:
> 
> 1) no plan survives first encounter with the enemy.  Parsoid was going to
> be simpler than the PHP parser, Parsoid was going to be written in PHP,
> then C, then prototyped in JS for a later implementation in C, etc.  It has
> varied over time as we learned more about the problem.  It is currently
> written in node.js and probably is at least the same order of complexity as
> the existing PHP parser.  It is, however, built on slightly more solid
> foundations, so its behavior is more regular than the PHP parser in many
> places -- although I've been submitting patches to the core parser where
> necessary to try to bring them closer together.  (c.f.
> https://gerrit.wikimedia.org/r/180982 for the most recent of these.)  And,
> of course, Parsoid emits well-formed HTML which can be round-tripped.
> 
> In many cases Parsoid could be greatly simplified if we didn't have to
> maintain compatibility with various strange corner cases in the PHP parser.
> 
> 2) Parsoid contains a partial implementation of the PHP expandtemplates
> module.  It was decided (I think wisely) that we didn't really gain
> anything by trying to reimplement this on the Parsoid side, though, and it
> was better to use the existing PHP code via api.php.  The alternative would
> be to basically reimplement quite a lot of mediawiki (lua embedding, the
> various parser functions extensions, etc) in node.js.  This *could* be done
> -- there is no technical reason why it cannot -- but nobody thinks it's a
> good idea to spend time on right now.
> 
> But the expandtemplates stuff basically works.   As I said, it doesn't
> contain all the crazy extensions that we use on the main WMF sites, but it
> would be reasonable to turn it on for a smaller stock mediawiki instance.
> In that sense it *could* be a full replacement for the Parser.
> 
> But note that even as a full parser replacement Parsoid depends on the PHP
> API in a large number of ways: imageinfo, siteinfo, language information,
> localized keywords for images, etc.  The idea of "independence" is somewhat
> vague.
>   --scott
> 
> 
> On Mon, Jan 19, 2015 at 11:58 PM, MZMcBride  wrote:
> 
> > Matthew Flaschen wrote:
> > >On 01/19/2015 08:15 AM, MZMcBride wrote:
> > >> And from this question flows another: why is Parsoid
> > >> calling MediaWiki's api.php so regularly?
> > >
> > >I think it uses it for some aspects of templates and hooks.  I'm sure
> > >the Parsoid team could explain further.
> >
> > I've been discussing Parsoid a bit and there's apparently an important
> > distinction between the preprocessor(s) and the parser. Though in practice
> > I think "parser" is used pretty generically. Further notes follow.
> >
> > I'm told in Parsoid,  and {{!}} are special-cased, while most other
> > parser functions require using the expandtemplates module of MediaWiki's
> > api.php. As I understand it, calling out to api.php is intended to be a
> > permanent solution (I thought it might be a temporary shim).
> >
> > If the goal was to just add more verbose markup to parser output, couldn't
> > we just have done that (in PHP)? Node.js was chosen over PHP due to
> > speed/performance considerations and concerns, from what I now understand.
> >
> > The view that Parsoid is going to replace the PHP parser seems to be
> > overly simplistic and goes back to the distinction between the parser and
> > preprocessor. Full wikitext transformation seems to require a preprocessor.
> >
> > MZMcBride
> >
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> 
> 
> 
> -- 
> (http://cscott.net)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
  
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Greg Grossmeier]

> Quick update:
> 
> I've had a great experience working on our mobile apps, but it's time to
> get back to core MediaWiki and help "clean my own house"... now that we've
> got Mobile Apps fully staffed I'm leaving the mobile department and will be
> reporting directly to Damon within WMF Engineering.

Great news Brion!

I'm excited about what your energy will bring (back) to this part of the
software stack.

And I'll remain reserved and not make fanboy comments; professionalism
and all.

Greg

PS: I remember seeing Brion at a Wikimania and only have the courage to
go up to him and say "You don't know me but I know you. Thanks for
everything you've done." Hadn't talked in-person again until I was
hired. 

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Gabriel Wicke]

Brion,

I'm glad to have you back on the dark server side!

This is great news for our APIs and the technical evolution of our stack in
general.

Cheers!

Gabriel

On Tue, Jan 20, 2015 at 9:12 AM, Daniel Kinzler 
wrote:

> Am 20.01.2015 um 17:34 schrieb Brion Vibber:
> > Quick update:
> >
> > I've had a great experience working on our mobile apps, but it's time to
> > get back to core MediaWiki and help "clean my own house"... now that
> we've
> > got Mobile Apps fully staffed I'm leaving the mobile department and will
> be
> > reporting directly to Damon within WMF Engineering.
>
> Hey Brion!
>
> To me, as a member of the Architecture Committee and someone often
> frustrated
> with the encrusted innards of MediaWiki, this is great news!
>
> Or as you might say: Awesome!
>
> Cheers,
> Daniel
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Erik Moeller]

Hooray! :-) You'll do great things, as always. Look forward to your
focused leadership in this area.

Erik
-- 
Erik Möller
VP of Product & Strategy, Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Tei]

Haha..  great.

Wellcome back Brion.

I am just a lurker on this mail list, but your post are always  so full of
energy. I love if that energy is the energy of mediawiki.

I hope that energy inspire others to not just the day to day process but
the crazy things you mention.  I remember when the idea of a wiki (a
website any visitor can edit?!) was totally freak, alien to everyone, I
hope projects like MediaWiki still mantain a open door to things that can
freak us again.


-- 
--
ℱin del ℳensaje.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Daniel Kinzler]

Am 20.01.2015 um 17:34 schrieb Brion Vibber:
> Quick update:
> 
> I've had a great experience working on our mobile apps, but it's time to
> get back to core MediaWiki and help "clean my own house"... now that we've
> got Mobile Apps fully staffed I'm leaving the mobile department and will be
> reporting directly to Damon within WMF Engineering.

Hey Brion!

To me, as a member of the Architecture Committee and someone often frustrated
with the encrusted innards of MediaWiki, this is great news!

Or as you might say: Awesome!

Cheers,
Daniel


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] I haven't looked lately...

[James Forrester]

On 19 January 2015 at 05:43, David Gerard  wrote:

> On 19 January 2015 at 01:46, Jay Ashworth  wrote:
>
> > Aha!  Sorry; I was behind on Parsoid, was the problem.  Yeah, if there's
> > a way to edit in MWtext, both for humans and programs, then that serves
> the
> > use cases I would be concerned about.  Thanks for the prompt
> clarification,
> > Daniel.
>
> There's also fourteen years of legacy page versions which I think
> absolutely nobody is going to convert.
>

​Code is very good at "just doing it" if we wanted to do it.​ A few billion
revisions are just ephemeral details if it actually makes sense to change
over. Not that we should do it blindly, but this shouldn't hold us back.

​J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforres...@wikimedia.org | @jdforrester
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Dan Garry]

On 20 January 2015 at 08:43, Siebrand Mazeland  wrote:

> This is fantastic news, although I fear Mobile may miss you at times.
>

We absolutely will.

However, we are excited that Brion is placing himself somewhere where he
can continue to be our ally and push forward the infrastructure that
Mobile, like many of us, are dependent on to succeed.

Enjoy yourself, Brion!

Dan

-- 
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Brion's role change within WMF

[Siebrand Mazeland]

This is fantastic news, although I fear Mobile may miss you at times. Enjoy
the (re)new(ed) area
 of work, Brion!

Siebrand

On Tue, Jan 20, 2015 at 5:34 PM, Brion Vibber  wrote:

> Quick update:
>
> I've had a great experience working on our mobile apps, but it's time to
> get back to core MediaWiki and help "clean my own house"... now that we've
> got Mobile Apps fully staffed I'm leaving the mobile department and will be
> reporting directly to Damon within WMF Engineering.
>
> First -- huge thanks to Monte and Dan and Kristen and Dmitry and Bernd and
> of course Tomasz!! and everybody else who's been awesome in Mobile Apps --
> and also to the rest of the mobile team, who have become too many to list
> in a concise email. :)
>
>
> For the moment I'm going to get back up to speed with the Architecture
> Committee and push at general MediaWiki issues. As we determine the fate of
> committees and narrow down what are our priority projects, my focus may
> narrow a to getting some particular things done over the next months.
>
>
> A few general notes:
>
> * Working in mobile apps reminded me how important our APIs are -- our
> ability to make flexible interfaces that work in different form factors and
> different technology stacks is dependent on maintaining a good API. This
> needs work. :)
>
> This doesn't just mean interacting with "api.php" -- we need clean
> configuration, data, etc interfaces as well, especially if we want people
> to contribute in ways other than raw text editing. There's a lot to clean
> up...
>
> * Mobile mobile mobile! I've heard some folks complain that while there's a
> lot of talk about "mobile-first" and similar there aren't always concrete
> explanations yet of what that means. I hope to bring some of the excitement
> we've seen in Mobile about Wikidata, better queries, better visual/user
> interaction design, and generally making things *work for users*.
>
> * Breaking or working around the "PHP barrier" for third-party MediaWiki
> users: I hope to get the question of services resolved one way or another
> -- either by us officially dropping "shared PHP hosting" support or by
> making sure we have "pure PHP" implementations of things that are required
> to operate -- which is mostly dependent on having good interfaces and APIs
> so that multiple implementations can be written and maintained
> compatibly...
>
> * Future stuff -- new media types, narrow-field editing, natural language
> queries? WHO KNOWS! I'll be researching more crazy stuff in my additional
> time.
>
>
> I'll see many of you at the Dev Summit soon enough -- don't be shy about
> pestering me with concerns and ideas about priorities. :)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Brion's role change within WMF

[Brion Vibber]

Quick update:

I've had a great experience working on our mobile apps, but it's time to
get back to core MediaWiki and help "clean my own house"... now that we've
got Mobile Apps fully staffed I'm leaving the mobile department and will be
reporting directly to Damon within WMF Engineering.

First -- huge thanks to Monte and Dan and Kristen and Dmitry and Bernd and
of course Tomasz!! and everybody else who's been awesome in Mobile Apps --
and also to the rest of the mobile team, who have become too many to list
in a concise email. :)


For the moment I'm going to get back up to speed with the Architecture
Committee and push at general MediaWiki issues. As we determine the fate of
committees and narrow down what are our priority projects, my focus may
narrow a to getting some particular things done over the next months.


A few general notes:

* Working in mobile apps reminded me how important our APIs are -- our
ability to make flexible interfaces that work in different form factors and
different technology stacks is dependent on maintaining a good API. This
needs work. :)

This doesn't just mean interacting with "api.php" -- we need clean
configuration, data, etc interfaces as well, especially if we want people
to contribute in ways other than raw text editing. There's a lot to clean
up...

* Mobile mobile mobile! I've heard some folks complain that while there's a
lot of talk about "mobile-first" and similar there aren't always concrete
explanations yet of what that means. I hope to bring some of the excitement
we've seen in Mobile about Wikidata, better queries, better visual/user
interaction design, and generally making things *work for users*.

* Breaking or working around the "PHP barrier" for third-party MediaWiki
users: I hope to get the question of services resolved one way or another
-- either by us officially dropping "shared PHP hosting" support or by
making sure we have "pure PHP" implementations of things that are required
to operate -- which is mostly dependent on having good interfaces and APIs
so that multiple implementations can be written and maintained compatibly...

* Future stuff -- new media types, narrow-field editing, natural language
queries? WHO KNOWS! I'll be researching more crazy stuff in my additional
time.


I'll see many of you at the Dev Summit soon enough -- don't be shy about
pestering me with concerns and ideas about priorities. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Getting the full URL of an image

[Brian Wolff]

On 1/19/15, Jeroen De Dauw  wrote:
> Hey,
>
> On my local wiki I have a page with the name "File:Blue marker.png". The
> following code returns false:
>
> $title = Title::makeTitle( NS_FILE, $file );
> $title->exists();
>
> That used to return true in the past. Not sure what is broken - my wiki or
> MediaWiki itself.
>

You're sure about that? I think you're just mixing up $title->exists()
and $title->isKnown().

>$title = Title::makeTitle( NS_FILE, $filename );
>$file = wfLocalFile( $title );
>$file->getUrl();

That's generally right, but depending on the exact context, the
following might be more appropriate:

$title = Title::makeTitle( NS_FILE, $filename );
$file = wfFindFile( $title );
$file->getFullUrl(); // or $file->getCanonicalUrl(); if you need the protocol.

wfLocalFile will always return a file object even if there is no file
by that name (And it will take its best guess at the url even for
non-existent files). If the name given would normally be a foreign
file, wfLocalFile( 'Foo.jpg' )->getUrl() would return what the url
would be if there was a local file of that name. On the other hand,
wfFindFile will return the appropriate (possibly foreign) file for
that name or false if none exists.

In almost all cases you would probably want the wfFindFile behavior
and not wfLocalFile().

--bawolff

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Parsoid's progress

[C. Scott Ananian]

I believe Subbu will follow up with a more complete response, but I'll note
that:

1) no plan survives first encounter with the enemy.  Parsoid was going to
be simpler than the PHP parser, Parsoid was going to be written in PHP,
then C, then prototyped in JS for a later implementation in C, etc.  It has
varied over time as we learned more about the problem.  It is currently
written in node.js and probably is at least the same order of complexity as
the existing PHP parser.  It is, however, built on slightly more solid
foundations, so its behavior is more regular than the PHP parser in many
places -- although I've been submitting patches to the core parser where
necessary to try to bring them closer together.  (c.f.
https://gerrit.wikimedia.org/r/180982 for the most recent of these.)  And,
of course, Parsoid emits well-formed HTML which can be round-tripped.

In many cases Parsoid could be greatly simplified if we didn't have to
maintain compatibility with various strange corner cases in the PHP parser.

2) Parsoid contains a partial implementation of the PHP expandtemplates
module.  It was decided (I think wisely) that we didn't really gain
anything by trying to reimplement this on the Parsoid side, though, and it
was better to use the existing PHP code via api.php.  The alternative would
be to basically reimplement quite a lot of mediawiki (lua embedding, the
various parser functions extensions, etc) in node.js.  This *could* be done
-- there is no technical reason why it cannot -- but nobody thinks it's a
good idea to spend time on right now.

But the expandtemplates stuff basically works.   As I said, it doesn't
contain all the crazy extensions that we use on the main WMF sites, but it
would be reasonable to turn it on for a smaller stock mediawiki instance.
In that sense it *could* be a full replacement for the Parser.

But note that even as a full parser replacement Parsoid depends on the PHP
API in a large number of ways: imageinfo, siteinfo, language information,
localized keywords for images, etc.  The idea of "independence" is somewhat
vague.
  --scott


On Mon, Jan 19, 2015 at 11:58 PM, MZMcBride  wrote:

> Matthew Flaschen wrote:
> >On 01/19/2015 08:15 AM, MZMcBride wrote:
> >> And from this question flows another: why is Parsoid
> >> calling MediaWiki's api.php so regularly?
> >
> >I think it uses it for some aspects of templates and hooks.  I'm sure
> >the Parsoid team could explain further.
>
> I've been discussing Parsoid a bit and there's apparently an important
> distinction between the preprocessor(s) and the parser. Though in practice
> I think "parser" is used pretty generically. Further notes follow.
>
> I'm told in Parsoid,  and {{!}} are special-cased, while most other
> parser functions require using the expandtemplates module of MediaWiki's
> api.php. As I understand it, calling out to api.php is intended to be a
> permanent solution (I thought it might be a temporary shim).
>
> If the goal was to just add more verbose markup to parser output, couldn't
> we just have done that (in PHP)? Node.js was chosen over PHP due to
> speed/performance considerations and concerns, from what I now understand.
>
> The view that Parsoid is going to replace the PHP parser seems to be
> overly simplistic and goes back to the distinction between the parser and
> preprocessor. Full wikitext transformation seems to require a preprocessor.
>
> MZMcBride
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l