I'm going to take the bait and respond in part, to defend the teams and
projects I work with:
On Tue, Nov 8, 2016 at 2:47 PM, Rogol Domedonfors <domedonf...@gmail.com>
> they are summarised by the four words
> under-resourced, under-managed and under-performing*. The VE/Parsoid/Flow
> complex suffers from scope mismatch. As a vehicle for delivering a WYSIWYG
> editor and discussion board it is over-complex,
I'll stop here. I think it is poorly understood in the community how
complex wikitext markup has been allowed to grow over the decades it has
been under development. There *is no specification for wikitext*. We have
informal guides which omit most of the interesting corner cases, like, say,
priority between conflicting markup. Take a look at
http://spec.commonmark.org/ to see what a precise specification for a *much
simpler* markup language would look like. As you read through the cases in
that spec, consider that if you translated most of the examples into
wikitext, *literally no one knows what the expected output would be*. The
implementation is too organic, with bits glommed on ad-hoc, and we'd
probably have to run them through the parser to find out what happens:
there is no way for a human to reason about it.
So, yes: VE/Parsoid are very complex (I'll leave Flow for a little further
down, be patient). Part of this was a learning process: when I joined the
project in 2013 there was a perhaps-naïve sense that we could easily
implement a sort of "cleaned up wikitext" and migrate any markup that
broke. Over the intervening years (and in response to heated criticism of
Visual Editor when the "cleaned up wikitext" failed to exactly match
editors' expectations), Parsoid has grown and grown, until it has almost as
many barnacles as the core PHP parser.
This is not only a challenge for engineering. This is a challenge to the
community. In order to actually achieve simplicity in our parser and
editor we will need to make aggressive changes to our content markup in
order to reverse years (decades) of poorly-understood workarounds to buggy
or underspecified wikitext syntax. Yes, you can fault WMF management (and
I'll personally take some blame) for not aggressively pushing the community
into this action, but the community in its turn has been strongly resistant
to proposed changes in wikitext, which they characterize as "for the sake
of Visual Editor", not fully realizing the extent to which the obscurity of
the current wikitext syntax imperils the editability, archivability and
future usability of the content we are creating.
We need to tackle this challenge together. Only when we commit to
aggressively updating the wikitext used in our projects, will we be able to
drop the "unnecessary" complexity of the parser-editor implementation.
There are a number of different proposals for "wikitext 2.0", more than one
of which I expect will be the subject of sessions at the developer summit
this year (
But again: the engineers can't do it alone. It's not just an engineering
problem. We *started* with the engineering solution (simple parsoid/VE
with no compatibility barnacles) and had to spend the next three years in
further development because of social/human factors --- namely, every time
that VE or Parsoid "broke" something the solution was to complicate the
parser and editor, not to fix the wikitext. If we want to scale down
ongoing development, we need to tackle the community side of the larger
issue of wikitext maintenance and migration.
I think you actually agree: you finish your open letter with, "Your staff
need to engage with users as equals, not de haut en bas. The community is
here to work with you if you will let it." But this goes both ways -- as a
community we need to stop treating our existing content as sacrosanct, and
recognize that the community needs to accept change if they want
engineering to finish projects and move the platform forward.
> while VE and Flow are
> under-ambitious. Can these really be regarded as cutting edge in 2016?
Agreed. We have collaborative features being rolled out in VE now, but
that just brings us to the state of the art circa 2005/6, when Google Docs
(nee Writely) was written (or ~2009, when AbiWord gained its collaborative
features in version 2.8.0). Yes, we're late. But the actual
implementation of a collaborative editor has always been the "easy part"
(see https://www.mediawiki.org/wiki/Extension:TogetherJS for example). The
hard parts were:
1. Always generating exactly the wikitext a human editor would generate for
a given edit (which varies by projects).
2. Open social collaboration among tens of millions of users on WMF
projects (~30 million on enwiki alone), while respecting privacy and
I discussed #1 above. #2 is still an open problem -- not just for us, but
for the "state of the art". If you are aware of a similar WMF-scale
collaborative editor, do let me know! In particular, balancing openness,
privacy, anti-harassment and anti-vandalism is a hard problem. We are
working on this, but this will also require buy-in from the community. We
have to embrace, together, a real-time UX for Wikipedia without reflexively
pushing back just because "it's not how editing on Wikipedia used to be".
We probably need to be open to trying things, learning what works and what
doesn't, and then trying other things -- without blaming WMF for the
shortcomings of the first attempt. This was a proposed topic for the dev
summit, has some proposed sessions as well (
https://phabricator.wikimedia.org/T149663), and I've been leading sessions
at Wikimania on this since 2014 to encourage discussion in the community
Perhaps this is actually your point: we need the community and engineering
to work together. We need the community to embrace change, enabling
engineering to implement that change.
> correctly scoped they could and should have been delivered and finalised
> long ago, if not brought in from pre-existing open source projects.
Again, if we just adopted (say) markdown or pure-HTML for our content, we
could have used one of the existing editors. But instead the community
wanted something which looked just like traditional wikitext. If there's a
failure in vision there, it cuts both ways.
> The execution of the VE project has been lacklustre. Such a
> product should have been finished long since, and it was clearly
> inadequately resourced and managed.
Again, there's has been a *lot* of background work to make a project "look"
straightforward. Inadequately resourced I could agree with -- there was a
WMF reorg under the previous ED that reassigned most of VE's engineers. It
still doesn't really have enough resources to tackle an aggressive project
like collaborative editing; especially considering that "adequate"
resourcing would probably involve all parts of engineering to develop an
integrated UX for collaboration, not just resources in the editing team.
Calls to cut resources and "finish" VE don't help.
The current culture appears not
> to pritorise such issues as timeliness and grip.
I'd say that the current culture rewards short-term projects with
immediately realizable and quantifiable goals that are highly likely to be
successful. That's not a bad thing, per se, but you seem to be arguing for
the adoption of larger scale projects with "vision". WMF has become scared
of large scale projects after the community rejection of Visual Editor,
Media Viewer, and Flow. It has interpreted the community's reaction to
these projects as a directive to "think smaller" and concentrate on simple
reactions to user bug reports.
This is what led to VE/Parsoid spending years tweaking the support of
corner case legacy constructs and trying to generate output that looks more
and more like traditional wikitext, instead of "finishing" the project. If
you want projects to end cleanly, the community has to agree one a point at
which things are "done enough" and that future issues should be
accommodated by content changes or behavior changes, not by endless
refinements to the code.
> Finally, Workflow has
> already failed. In the absence of resources to undertake the required
> research into the huge complexity of work flows within the projects,
> whatever is designed will be designed in ignorance, and hence simply can
> not succeed. Ever. It is already a waste of time.
I think you are suggesting that we devote the resources required to
properly understand the workflow problem and adequately solve the problem?
What we are currently doing is actually what you indicate: we have stopped
development on Flow and given up on the general workflow problem, instead
looking at small tools for specific work flows -- like the addition of ORES
scores to various review processes. We are not currently working on the
general workflow problem.
> In terms of the wider project, the view of an editing and rendering engine
> as handling only unidirectional linear text fails to capture even the
> richness of current projects let alone future knowledge modalities, such as
> complex text (hieroglyphics, chemistry, mathematics, music),
> higher-dimensional data (genomic, proteomic, 3D printing), data in time
> (audio, video), interactive, computational, ... all of which could and
> should have been scoped out by your innovation work.
I'm not sure why you think that VE/Parsoid only supports unidirectional
linear text? Our RTL support is actually quite robust. And VE supports
WikiHiero -- see for example
Further extensions are limited mostly by the community's willingness to
write support: the infrastructure to add extensions to Visual Editor and
Parsoid are both present.
I do think we should spend time thinking about non-text content.
, https://phabricator.wikimedia.org/T149554, and
https://phabricator.wikimedia.org/T90914#2759122 for instance. I think the
general rule of Wikimedia Incubator projects holds -- if the community
generated really compelling content in new media, WMF engineering would be
increasingly motivated to provide tools to support that new media. It's a
bit of a chicken-and-egg problem. We tried to envision what "cool content"
might look like at the last Wikimania (see "Integrated video" and
"Integrated VR/panorama" at https://en.wikipedia.org/wiki/User:Cscott/Ideas),
but I think engineering is rightly reluctant to spend a lot of resources
developing these sorts of features w/o content requiring them.
To conclude, I would like to echo your call for increased collaboration
between community and engineering. But in addition to the "engineering
should build what the community wants" direction, we need to also build the
"community should embrace changes arising from WMF efforts to evolve the
platform" direction. The necessary balance and interplay between these
factors has been missing, and that has resulted in an engineering
department which is too under-resourced to actually execute a vision for
the future. And in the absence of a vision for the future, the community
has called for engineering to scale down (further) and "focus".
This is not healthy for the future of the project.
Wikimedia-l mailing list, guidelines at:
New messages to: Wikimediafirstname.lastname@example.org