[Wikitech-l] Technical leadership in Wikimedia
Hi all, your thoughts would be appreciated on this. It would be great to get some input from tech-focused contributors. I started the thread only on Wikimedia-l to keep the discussion consolidated in one place. http://lists.wikimedia.org/pipermail/wikimedia-l/2014-May/071811.html Thanks, Pine ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Deployments of new, radically different default features
I said this during the retrospective about beta features. I think all beta features should be enabled for all logged in users the month before they get deployed. Let's not resort to banners please. Let's instead train people to go to the beta features page and help us build great products that meet both user and administrator needs upon first release. On 18 May 2014 19:24, Rainer Rillke rainerril...@hotmail.com wrote: Sun May 18 00:20:10 UTC 2014 Brian Wolff bawolff at gmail.com wrote: I do think there are probably better ways to handle notification of new features. Perhaps a pop up the first time new feature is activated explaining the feature and how to disable it. I'm not really sure. Yeah, that would be cool: I am tool x, I do y and you can disable me pressing button z. Let button z be a prominent element of the UI for the time of testing at large scale. This way, we can get rid of some of the banners, thus reducing the risk for ~-blindness and some of the trouble. VE has something similar now, if I recall correctly (though the disable-me-button is missing). -- Rillke --- I also like MMV as a visitor; but it doesn't fit into any administrator's workflow at Commons most of the time. Perhaps you also want to make it easier for anyone to report issues? Other sites have heavy libraries allowing users to pick the trouble-maker, or take a screenshot and cutting it, ... ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
I'm getting the impression there is a fundamental misunderstanding here.
Am 18.05.2014 04:28, schrieb Subramanya Sastry:
So, consider this wikitext for page P.
== Foo ==
{{wikitext-transclusion}}
*a1
map .. ... /map
*a2
{{T}} (the html-content-model-transclusion)
*a3
Parsoid gets wikitext from the API for {{wikitext-transclusion}}, parses it
and
injects the tokens into the P's content. Parsoid gets HTML from the API for
map./map and injects the HTML into the not-fully-processed wikitext
of P
(by adding an appropriate token wrapper). So, if {{T}} returns HTML (i.e. the
MW
API lets Parsoid know that it is HTML), Parsoid can inject the HTML into the
not-fully-processed wikitext and ensure that the final output comes out right
(in this case, the HTML from both the map extension and {{T}} would not get
sanitized as it should be).
Does that help explain why we said we don't need the html wrapper?
No, it actually misses my point completely. My point is that this may work with
the way parsoid uses expandtemplates, but it does not work for expandtemplates
in general. Because expandtemplates takes full wikitext as input, and only
partially replaces it.
So, let me phrase it this way:
If expandtemplates is called with text=
== Foo ==
{{T}}
[[Category:Bla]]
What should it return, and what content type should be declared in the http
header?
Note that I'm not talking about how parsoid processes this text. That's not my
point - my point is that expandtemplates can be and is used on full wikitext. In
that context, the return type cannot be HTML.
All that said, if you want to provide the wrapper with html model=whatever
fully-expanded-HTML/html, we can handle that as well. We'll use the
model
attribute of the wrapper, discard the wrapper and use the contents in our
pipeline.
Why use the model attribute? Why would you care about the original model? All
you need to know is that you'll get HTML. Exposing the original model in this
context seems useless if not misleading. html transclude={{T}}/html would
give that backend parser a way to discard the HTML (as unsafe) and execute the
transclusion instead (generating trusted HTML). In fact, we could just omit the
content of the html tag.
So, model information either as an attribute on the wrapper, api response
header, or a property in the JSON/XML response structure would all work for
us.
As explained above, the return type cannot be HTML for the full text, because
any plain wikitext would stay unprocessed. There needs to be a marker for
html transclusion *here* in the text.
Am 18.05.2014 16:29, schrieb Gabriel Wicke:
The difference between wrapper and property is actually that using inline
wrappers in the returned wikitext would force us to escape similar wrappers
from normal template content to avoid opening a gaping XSS hole.
Please explain, I do not see the hole you mention.
If the input contained htmlevil stuff/html, it would just get escaped by the
preprocessor (unless $wgRawHtml is enabled), as it is now:
https://de.wikipedia.org/w/api.php?action=expandtemplatestext=%3Chtml%3E%3Cscript%3Ealert%28%27evil%27%29%3C/script%3E%3C/html%3E
If html transclude={{T}} was passed, the parser/preprocessor would treat it
like it would treat {{T}} - it would get trusted, backend generated HTML from
respective Content object.
I see no change, and no opportunity to inject anything. Am I missing something?
A separate property in the JSON/XML structure avoids the need for escaping
(and associated security risks if not done thoroughly), and should be
relatively straightforward to implement and consume.
As explained above, I do not see how this would work except for the very special
case of using expandtemplates to expand just a single template. This could be
solved by introducing a new, single template mode for expandtemplates, e.g.
using expand=Foo|x|y|z instead of text={{Foo|x|y|z}}.
Another way would be to use hints the structure returned by generatexml. There,
we have an opportunity to declare a content type for a *part* of the output (or
rather, input).
-- daniel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Reducing botspam on #wikimedia-dev
On May 18, 2014 9:36 PM, K. Peachey p858sn...@gmail.com wrote: tbh -dev (or somewhere else) should be the master channel for bz output. The somewhere else is #mediawiki-feed with master feeds for both gerrit and bugzilla. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Performance guidelines chat Monday (for Asia/Australia)
On 05/16/2014 07:11 PM, Sumana Harihareswara wrote: http://www.worldtimebuddy.com/?qm=1lid=1816670,2147714,1277333,5128581h=1816670date=2014-5-19sln=19-20 I will be in #wikimedia-dev on Monday for an hour to talk about the performance guidelines https://www.mediawiki.org/wiki/Performance_guidelines as well as upcoming security and architecture guidelines: New York: 7am Bangalore: 4:30pm Beijing: 7pm Sydney: 9pm This isn't one of the RfC meetings. :) This is starting at 11am UTC, in half an hour. -- Sumana Harihareswara Senior Technical Writer Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Language Engineering IRC Office Hour on May 21, 2014 (Wednesday) at 1700 UTC
[x-posted] Hello, The Wikimedia Language Engineering team will be hosting the next monthly IRC office hour on Wednesday, May 21 2014 at 1700 UTC on #wikimedia-office. The event is delayed this month as the team was traveling. In this office hour we will be discussing about our recent work, which has mostly been around the upcoming first release of the Content Translation tool[1]. We will also be taking questions during the session. Please see below for event details and local time. See you at the office hour. Thanks Runa [1] https://www.mediawiki.org/wiki/Content_translation Monthly IRC Office Hour: == # Date: May 21, 2014 (Wednesday) # Time: 1700 UTC/1000PDT (Check local time: http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140521T1700) # IRC channel: #wikimedia-office # Agenda: 1. Content Translation project updates 2. Q A (Questions can be sent to me ahead of the event) -- Language Engineering - Outreach and QA Coordinator Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Performance guidelines chat Monday (for Asia/Australia)
On 05/19/2014 06:30 AM, Sumana Harihareswara wrote: On 05/16/2014 07:11 PM, Sumana Harihareswara wrote: http://www.worldtimebuddy.com/?qm=1lid=1816670,2147714,1277333,5128581h=1816670date=2014-5-19sln=19-20 I will be in #wikimedia-dev on Monday for an hour to talk about the performance guidelines https://www.mediawiki.org/wiki/Performance_guidelines as well as upcoming security and architecture guidelines: New York: 7am Bangalore: 4:30pm Beijing: 7pm Sydney: 9pm This isn't one of the RfC meetings. :) This is starting at 11am UTC, in half an hour. Moving to #mediawiki which is a bit quieter. :) -- Sumana Harihareswara Senior Technical Writer Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 04:52 AM, Daniel Kinzler wrote:
I'm getting the impression there is a fundamental misunderstanding here.
You are correct. I completely misunderstood what you said in your last
response about expandtemplates. So, the rest of my response to your last
email is irrelevant ... and let me reboot :-).
All that said, if you want to provide the wrapper with html model=whatever
fully-expanded-HTML/html, we can handle that as well. We'll use the model
attribute of the wrapper, discard the wrapper and use the contents in our
pipeline.
Why use the model attribute? Why would you care about the original model? All
you need to know is that you'll get HTML. Exposing the original model in this
context seems useless if not misleading.
Given that I misunderstood your larger observation about
expandtemplates, this is not relevant now. But, I was basing this on
your proposal from the previous email which I'll now go back to.
On 05/17/2014 06:14 PM, Daniel Kinzler wrote:
I think something like html transclusion={{T}}
model=whatever.../html would work best.
I see what you are getting at here. Parsoid can treat this like a
regular tag-extension and send it back to the api=parse endpoint for
processing. Except if you provided the full expansion as the content of
the html-wrapper in which case the extra api call can be skipped. The
extra api call is not really an issue for occasional uses, but on pages
with a lot of non-wikitext transclusion uses, this is an extra api call
for each such use. I don't have a sense for how common this would be, so
maybe that is a premature worry.
That said, for other clients, this content would be deadweight (if they
are going to discard it and go back to the api=parse endpoint anyway or
worse send back the entire response to the parser that is going to just
discard it after the network transfer).
So, looks like there are some conflicting perf. requirements for
different clients wrt expandtemplates response here. In that context, at
least from a solely parsoid-centric point of view, the new api endpoint
'expand=Foo|x|y|z' you proposed would work well as well.
Subbu.
A separate property in the JSON/XML structure avoids the need for escaping
(and associated security risks if not done thoroughly), and should be
relatively straightforward to implement and consume.
As explained above, I do not see how this would work except for the very special
case of using expandtemplates to expand just a single template. This could be
solved by introducing a new, single template mode for expandtemplates, e.g.
using expand=Foo|x|y|z instead of text={{Foo|x|y|z}}.
Another way would be to use hints the structure returned by generatexml. There,
we have an opportunity to declare a content type for a *part* of the output (or
rather, input).
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Deployments of new, radically different default features
On 19 May 2014 03:49, Jon Robson jdlrob...@gmail.com wrote: I said this during the retrospective about beta features. I think all beta features should be enabled for all logged in users the month before they get deployed. Let's not resort to banners please. This is a much better idea than putting up banners which people learn to tune out. For me, not only do I not read them anymore, but it's gotten to the point now that I don't even notice whether there's even a banner up. If we were to do this, we should make sure that people are aware upon opting out of the beta feature that their preference will only be respected until it's graduated. Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Deployments of new, radically different default features
On Sun, May 18, 2014 at 08:22:13PM +0200, Rainer Rillke wrote: Yeah, that would be cool: I am tool x, I do y and you can disable me pressing button z. Let button z be a prominent element of the UI for the time of testing at large scale. For the record, we did the first part of this - there was a link to the preferences page that would let you disable the feature at the bottom of the metadata panel from the day we pushed to the 3rd pilot sites, I think. As for prominent, I don't think that's a good idea because it would disrupt the realism of the feature. The purpose of such a feature would be temporary, and when it got removed from the UI, it would cause a (small) jolt to users. We already have a bunch of weird one-time things that are cluttering the toolbar, we didn't need another one disrupting the flow of our product. If you're a power user, you know where Special:Preferences is, and we made sure to help you out as much as possible if you don't. I don't think we needed to do anything to make the preference more discoverable, it would have been a waste of our time to do so given the other things we have on our plates. I can't speak to the community side of things - I was under the impression that very few people had actually had that much trouble with the docs, but perhaps the issues with them should be brought up *on their talk page* rather than launching some weird campaign over a feature that, frankly, could not *possibly* have been launched in a more cautious way. Cheers, -- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtrac...@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist signature.asc Description: Digital signature ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
Am 19.05.2014 14:21, schrieb Subramanya Sastry:
On 05/19/2014 04:52 AM, Daniel Kinzler wrote:
I'm getting the impression there is a fundamental misunderstanding here.
You are correct. I completely misunderstood what you said in your last
response
about expandtemplates. So, the rest of my response to your last email is
irrelevant ... and let me reboot :-).
Glad we got that out of the way :)
On 05/17/2014 06:14 PM, Daniel Kinzler wrote:
I think something like html transclusion={{T}} model=whatever.../html
would work best.
I see what you are getting at here. Parsoid can treat this like a regular
tag-extension and send it back to the api=parse endpoint for processing.
Except
if you provided the full expansion as the content of the html-wrapper in which
case the extra api call can be skipped. The extra api call is not really an
issue for occasional uses, but on pages with a lot of non-wikitext
transclusion
uses, this is an extra api call for each such use. I don't have a sense for
how
common this would be, so maybe that is a premature worry.
I would probably go for always including the expanded HTML for now.
That said, for other clients, this content would be deadweight (if they are
going to discard it and go back to the api=parse endpoint anyway or worse send
back the entire response to the parser that is going to just discard it after
the network transfer).
Yes. There could be an option to omit it. That makes the implementation more
complex, but it's doable.
So, looks like there are some conflicting perf. requirements for different
clients wrt expandtemplates response here. In that context, at least from a
solely parsoid-centric point of view, the new api endpoint 'expand=Foo|x|y|z'
you proposed would work well as well.
That seems the cleanest solution for the parsoid use case - however, the
implementation is complicated by how parameter substitution works. For HTML
based transclusion, it doesn't work at all at the moment - we would need tighter
integration with the preprocessor for doing that.
Basically, there would be two cases: convert expand=Foo|x|y|z to {{Foo|x|y|z}}
internally an call Parser::preprocess on that, so parameter subsitution is done
correctly; or get the HTML from Foo, and discard the parameters. We would have
to somehow know in advance which mode to use, handle the appropriate case, and
then set the Content-Type header accordingly. Pretty messy...
I think html transclusion={{T}} is the simplest and most robust solution for
now.
-- daniel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Status of the new PDF Renderer
Hi, On 01/18/2014 03:42 AM, Matthew Walker wrote: We've just finished our second sprint on the new PDF renderer. A significant chunk of renderer development time this cycle was on non latin script support, as well as puppetization and packaging for deployment. We have a work in progress pipeline up and running in labs which I encourage everyone to go try and break. Seeing breakage of PDF downloads on pl.wikisource reported in https://bugzilla.wikimedia.org/show_bug.cgi?id=65298 I got curious what the status of the new PDF renderer is. https://www.mediawiki.org/wiki/PDF_rendering#Status only links to the quoted email from January 2014. Has anything happened in the last four months that is worth to be added as a status update? For the records, Nemo asked on the Talk page for a test instance and I support the idea of having a bugday on PDF rendering once public testing infrastructure for the new PDF renderer is available. Open tickets to potentially re-test: https://bugzilla.wikimedia.org/buglist.cgi?resolution=---component=Collection andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 09:52 AM, Daniel Kinzler wrote: Am 18.05.2014 16:29, schrieb Gabriel Wicke: The difference between wrapper and property is actually that using inline wrappers in the returned wikitext would force us to escape similar wrappers from normal template content to avoid opening a gaping XSS hole. Please explain, I do not see the hole you mention. If the input contained htmlevil stuff/html, it would just get escaped by the preprocessor (unless $wgRawHtml is enabled), as it is now: https://de.wikipedia.org/w/api.php?action=expandtemplatestext=%3Chtml%3E%3Cscript%3Ealert%28%27evil%27%29%3C/script%3E%3C/html%3E What you see there is just unescaped HTML embedded in the XML result format. It's clearer that there's in fact no escaping on the HTML when looking at the JSON: https://de.wikipedia.org/w/api.php?action=expandtemplatestext=%3Chtml%3E%3Cscript%3Ealert%28%27evil%27%29%3C/script%3E%3C/html%3Eformat=json Parsoid depends on there being no escaping for unknown tags (and known extension tags) in the preprocessor. So if you use tags, you'll have to add escaping for those. The move to HTML-based (self-contained) transclusions expansions will avoid this issue completely. That's a few months out though. Maybe we can find a stop-gap solution that moves in that direction, without introducing special tags in expandtemplates that we'll have to support for a long time. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 04:54 PM, Gabriel Wicke wrote:
The move to HTML-based (self-contained) transclusions expansions will avoid
this issue completely. That's a few months out though. Maybe we can find a
stop-gap solution that moves in that direction, without introducing special
tags in expandtemplates that we'll have to support for a long time.
Here's a proposal:
* Introduce a domparse extension tag that causes its content to be parsed
all the way to a self-contained DOM structure. Example:
domparse{{T}}/domparse
* Emit this tag for HTML page transclusions. Avoids the security issue as
there's no way to inject verbatim HTML. Works with Parsoid out of the box.
* Use domparse to support parsing unbalanced templates by inserting it
into wikitext:
domparse
{{table-start}}
{{table-row}}
{{table-end}}
/domparse
* Build a solid HTML-only expansion API end point, and start using that for
all transclusions that are not wrapped in domparse
* Stop wrapping non-wikitext transclusions into domparse in
action=expandtemplates once those can be directly expanded to a
self-contained DOM.
Gabriel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 10:19 AM, Gabriel Wicke wrote:
On 05/19/2014 04:54 PM, Gabriel Wicke wrote:
The move to HTML-based (self-contained) transclusions expansions will avoid
this issue completely. That's a few months out though. Maybe we can find a
stop-gap solution that moves in that direction, without introducing special
tags in expandtemplates that we'll have to support for a long time.
Here's a proposal:
* Introduce a domparse extension tag that causes its content to be parsed
all the way to a self-contained DOM structure. Example:
domparse{{T}}/domparse
* Emit this tag for HTML page transclusions. Avoids the security issue as
there's no way to inject verbatim HTML. Works with Parsoid out of the box.
* Use domparse to support parsing unbalanced templates by inserting it
into wikitext:
domparse
{{table-start}}
{{table-row}}
{{table-end}}
/domparse
* Build a solid HTML-only expansion API end point, and start using that for
all transclusions that are not wrapped in domparse
* Stop wrapping non-wikitext transclusions into domparse in
action=expandtemplates once those can be directly expanded to a
self-contained DOM.
Here's a possible division of labor:
You (Daniel) could start with the second step (emitting the tag). Since not
much escaping is needed (only nested domparse tags in the transclusion)
this should be fairly straightforward.
We could work on the extension implementation (first bullet point) together,
or tackle it completely on the Parsoid side. We planned to work on this in
any case as part of our longer-term migration to well-balanced HTML
transclusions.
The advantage of using domparse to support both unbalanced templates
special transclusions is that we'll only have to implement this once, and
won't introduce another tag only to deprecate it fairly quickly. Phasing out
unbalanced templates will take longer, as we'll first have to come up with
alternative means to support the same use cases.
Gabriel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 10:55 AM, Bartosz Dziewoński wrote: I am kind of lost in this discussion, but let me just ask one question. Won't all of the proposed solutions, other than the one of just not expanding transclusions that can't be expanded to wikitext, break the original and primary purpose of ExpandTemplates: providing valid parsable wikitext, for understanding by humans and for pasting back into articles in order to bypass transclusion limits? Yup. But that's the case with domparse, while it's not the case with html unless $wgRawHtml is true (which is impossible for publicly-editable wikis). I feel that Parsoid should be using a separate API for whatever it's doing with the wikitext. I'm sure that would give you more flexibility with internal design as well. We are moving towards that, but will still need to support unbalanced transclusions for a while. Since special transclusions can be nested inside of those we will need some form of inline support even if we expand most transclusions all the way to DOM with a different end point. Also, as Daniel pointed out, most other users are using action=expandtemplates for entire pages and expect that to work as well. Gabriel ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
I am kind of lost in this discussion, but let me just ask one question. Won't all of the proposed solutions, other than the one of just not expanding transclusions that can't be expanded to wikitext, break the original and primary purpose of ExpandTemplates: providing valid parsable wikitext, for understanding by humans and for pasting back into articles in order to bypass transclusion limits? I feel that Parsoid should be using a separate API for whatever it's doing with the wikitext. I'm sure that would give you more flexibility with internal design as well. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Status of the new PDF Renderer
That's a good question! I'm in SFO this week, so it's probably worth setting aside a day to resync and figure out what the next steps for the new PDF renderer are. --scott ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
Am 19.05.2014 20:01, schrieb Gabriel Wicke:
On 05/19/2014 10:55 AM, Bartosz Dziewoński wrote:
I am kind of lost in this discussion, but let me just ask one question.
Won't all of the proposed solutions, other than the one of just not
expanding transclusions that can't be expanded to wikitext, break the
original and primary purpose of ExpandTemplates: providing valid parsable
wikitext, for understanding by humans and for pasting back into articles in
order to bypass transclusion limits?
Yup. But that's the case with domparse, while it's not the case with
html unless $wgRawHtml is true (which is impossible for publicly-editable
wikis).
html transclusion={{T}} would work transparently. It would contain HTML, for
direct use by the client, and could be passed back to the parser, which would
ignore the HTML and execute the transclusion. It should be 100% compatible with
existing clients (unless the look for verbatim html for some reason).
I'll have to re-read Gabriel's domparse proposal tomorrow - right now, I don't
see why it would be necessary, or how it would improve the situation.
I feel that Parsoid should be using a separate API for whatever it's doing
with the wikitext. I'm sure that would give you more flexibility with
internal design as well.
We are moving towards that, but will still need to support unbalanced
transclusions for a while.
But for HTML based transclusions you could ignore that - you could already
resolve these using a separate API call, if needed.
But still - I do not see why that would be necessary. If expandtemplates returns
html transclusion={{T}}, clients can pass that back to the parser safely, or
use the contained HTML directly, safely.
Parsoid would keep working as before: it would treat html as a tag extension
(it does that, right?) and pass it back to the parser (which would expand it
again, this time fully, if action=parse is used). If parsoid knows about the
special properties of html, it could just use the contents verbatim - I see no
reason why that would be any more unsafe as any other HTML returned by the
parser.
But perhaps I'm missing something obvious. I'll re-read the proposal tomorrow.
-- daniel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages
On 05/19/2014 12:46 PM, Daniel Kinzler wrote:
Am 19.05.2014 20:01, schrieb Gabriel Wicke:
On 05/19/2014 10:55 AM, Bartosz Dziewoński wrote:
I am kind of lost in this discussion, but let me just ask one question.
Won't all of the proposed solutions, other than the one of just not
expanding transclusions that can't be expanded to wikitext, break the
original and primary purpose of ExpandTemplates: providing valid parsable
wikitext, for understanding by humans and for pasting back into articles in
order to bypass transclusion limits?
Yup. But that's the case with domparse, while it's not the case with
html unless $wgRawHtml is true (which is impossible for publicly-editable
wikis).
html transclusion={{T}} would work transparently. It would contain HTML,
for
direct use by the client, and could be passed back to the parser, which would
ignore the HTML and execute the transclusion. It should be 100% compatible
with
existing clients (unless the look for verbatim html for some reason).
Currently html tags are escaped when $wgRawHtml is disabled. We could
change the implementation to stop doing so *iff* the transclusion parameter
is supplied, but IMO that would be fairly unexpected and inconsistent behavior.
I feel that Parsoid should be using a separate API for whatever it's doing
with the wikitext. I'm sure that would give you more flexibility with
internal design as well.
We are moving towards that, but will still need to support unbalanced
transclusions for a while.
But for HTML based transclusions you could ignore that - you could already
resolve these using a separate API call, if needed.
Yes, and they are going to be the common case once we have marked up the
exceptions with tags like domparse. As you correctly pointed out, inline
tags are primarily needed for expandtemplates calls on compound content,
which we need to do as long as we support unbalanced templates. We can't
know a priori whether some transclusions in turn transclude special HTML
content.
I think we have agreement that some kind of tag is still needed. The main
point still under discussion is on which tag to use, and how to implement
this tag in the parser.
Originally, domparse was conceived to be used in actual page content to
wrap wikitext that is supposed to be parsed to a balanced DOM *as a unit*
rather than transclusion by transclusion. Once unbalanced compound
transclusion content is wrapped in domparse tags (manually or via bots
using Parsoid info), we can start to enforce nesting of all other
transclusions by default. This will make editing safer and more accurate,
and improve performance by letting us reuse expansions and avoid
re-rendering the entire page during refreshLinks. See
https://bugzilla.wikimedia.org/show_bug.cgi?id=55524 for more background.
The use of domparse to mark up special HTML transclusions in
expandtemplates output will be temporary (until HTML transclusions are the
default), but even if such output is pasted into the actual wikitext it
would be harmless, and would work as expected.
Now back to the syntax. Encoding complex transclusions in a HTML parameter
would be rather cumbersome, and would entail a lot of attribute-specific
escaping. Wrapping such transclusions in domparse tags on the other hand
normally does not entail any escaping, as only nested domparse tags are
problematic.
Parsoid would keep working as before: it would treat html as a tag extension
(it does that, right?)
$wgRawHtml is disabled in all wikis we are currently interested in.
MediaWiki does properly report the html extension tag from siteinfo when
$wgRawHtml is enabled, so it ought to work with Parsoid for private wikis.
It will be harder to support the html
transclusion=transclusions/html exception.
Gabriel
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Bot flags and human-made edits
Hi everyone, I'm trying to figure out the reason behind some decisions that were made in the past about bot flags to see if we can have a more optimal and clear setup. Presently, giving an account the bot flag does two things: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. 2. When editing via the standard editing interface, flags all edits (i.e. all human made edits) as bot edits. If you've not got the bot flag, the API will ignore you if you try to flag an edit as a bot edit using the bot parameter. So I've got a few questions to help me figure this out. 1. What's the user story for including the edit-level granularity for bot accounts in the API? 2. What's the user story for making it so that every edit made by a human on a bot account is flagged as bot edit? Thanks, Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
Actually there are a few cases in the non API where bots can assert not being a bot, and there are some cases where non-bots can flag as bots for specific cases (I know it in the past it was used to suppress RC floods of mass vandalism reverts by admins) so your picture isnt complete On Mon, May 19, 2014 at 7:09 PM, Dan Garry dga...@wikimedia.org wrote: Hi everyone, I'm trying to figure out the reason behind some decisions that were made in the past about bot flags to see if we can have a more optimal and clear setup. Presently, giving an account the bot flag does two things: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. 2. When editing via the standard editing interface, flags all edits (i.e. all human made edits) as bot edits. If you've not got the bot flag, the API will ignore you if you try to flag an edit as a bot edit using the bot parameter. So I've got a few questions to help me figure this out. 1. What's the user story for including the edit-level granularity for bot accounts in the API? 2. What's the user story for making it so that every edit made by a human on a bot account is flagged as bot edit? Thanks, Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
On Mon, May 19, 2014 at 4:09 PM, Dan Garry dga...@wikimedia.org wrote: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. I'm responsible for this part of the mess. I don't remember why it was done this way though. I think the people that I took over the API edit code from had put this feature in and I kept it. What I think was going on is that no one could be bothered to figure out how to represent I am a human editing using a bot account and I'd like this edit to not be bot-flagged in the UI, so it wasn't done. But in the API it was trivial (bot=1), so it was done. That's just my recollection of it and it's probably flawed, this was 6-7 years ago. Roan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
Can you help me out and tell me what those cases are? I've been editing for nine years and not stumbled upon them, so I'm very curious. Thanks, Dan On 19 May 2014 19:13, John phoenixoverr...@gmail.com wrote: Actually there are a few cases in the non API where bots can assert not being a bot, and there are some cases where non-bots can flag as bots for specific cases (I know it in the past it was used to suppress RC floods of mass vandalism reverts by admins) so your picture isnt complete On Mon, May 19, 2014 at 7:09 PM, Dan Garry dga...@wikimedia.org wrote: Hi everyone, I'm trying to figure out the reason behind some decisions that were made in the past about bot flags to see if we can have a more optimal and clear setup. Presently, giving an account the bot flag does two things: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. 2. When editing via the standard editing interface, flags all edits (i.e. all human made edits) as bot edits. If you've not got the bot flag, the API will ignore you if you try to flag an edit as a bot edit using the bot parameter. So I've got a few questions to help me figure this out. 1. What's the user story for including the edit-level granularity for bot accounts in the API? 2. What's the user story for making it so that every edit made by a human on a bot account is flagged as bot edit? Thanks, Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
Sorry, I mean, can you tell me specifically how that's done? I don't know that. Thanks, Dan On 19 May 2014 19:33, Dan Garry dga...@wikimedia.org wrote: Can you help me out and tell me what those cases are? I've been editing for nine years and not stumbled upon them, so I'm very curious. Thanks, Dan On 19 May 2014 19:13, John phoenixoverr...@gmail.com wrote: Actually there are a few cases in the non API where bots can assert not being a bot, and there are some cases where non-bots can flag as bots for specific cases (I know it in the past it was used to suppress RC floods of mass vandalism reverts by admins) so your picture isnt complete On Mon, May 19, 2014 at 7:09 PM, Dan Garry dga...@wikimedia.org wrote: Hi everyone, I'm trying to figure out the reason behind some decisions that were made in the past about bot flags to see if we can have a more optimal and clear setup. Presently, giving an account the bot flag does two things: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. 2. When editing via the standard editing interface, flags all edits (i.e. all human made edits) as bot edits. If you've not got the bot flag, the API will ignore you if you try to flag an edit as a bot edit using the bot parameter. So I've got a few questions to help me figure this out. 1. What's the user story for including the edit-level granularity for bot accounts in the API? 2. What's the user story for making it so that every edit made by a human on a bot account is flagged as bot edit? Thanks, Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
As a bot operator I think API parameter about flagging bot or not is necessary but I think the best solution would be having something like flag=1 (optional) and this causes bot edits to be marked as human and for admin (+flooders) edits marked as bot. Implementing the parameter (whether the current status quo or my suggestion) in UI is important, I think adding a mark box besides minor and watch options would be great (for bots, admins and flooders) Best On Tue, May 20, 2014 at 3:59 AM, Roan Kattouw roan.katt...@gmail.comwrote: On Mon, May 19, 2014 at 4:09 PM, Dan Garry dga...@wikimedia.org wrote: 1. When editing via the API, allows the user to choose whether or not to flag an edit as a bot edit using the bot parameter. I'm responsible for this part of the mess. I don't remember why it was done this way though. I think the people that I took over the API edit code from had put this feature in and I kept it. What I think was going on is that no one could be bothered to figure out how to represent I am a human editing using a bot account and I'd like this edit to not be bot-flagged in the UI, so it wasn't done. But in the API it was trivial (bot=1), so it was done. That's just my recollection of it and it's probably flawed, this was 6-7 years ago. Roan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Amir ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
On Mon, May 19, 2014 at 4:33 PM, Dan Garry dga...@wikimedia.org wrote: Can you help me out and tell me what those cases are? I've been editing for nine years and not stumbled upon them, so I'm very curious. If you are a sysop, you can either add bot=1 to a rollback URL by hand (unlike most other state-changing actions, rollbacks are done via GET requests so you can do them with a single click), or go to a user's contribution list and add bot=1 to that URL (which will in turn add it to every rollback URL on the page). The related userright is markbotedits. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
Admins have the ability to mark their rollbacks as bot edits? Wow, that's fascinating. Talk about edge cases. Thanks Gergő! Dan On 19 May 2014 19:51, Gergo Tisza gti...@wikimedia.org wrote: On Mon, May 19, 2014 at 4:33 PM, Dan Garry dga...@wikimedia.org wrote: Can you help me out and tell me what those cases are? I've been editing for nine years and not stumbled upon them, so I'm very curious. If you are a sysop, you can either add bot=1 to a rollback URL by hand (unlike most other state-changing actions, rollbacks are done via GET requests so you can do them with a single click), or go to a user's contribution list and add bot=1 to that URL (which will in turn add it to every rollback URL on the page). The related userright is markbotedits. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
On 19 May 2014 19:36, Amir Ladsgroup ladsgr...@gmail.com wrote: As a bot operator I think API parameter about flagging bot or not is necessary Sure, but as I'm not a bot operator, can you explain why and what you use this for, to help me understand? :-) Implementing the parameter (whether the current status quo or my suggestion) in UI is important, I think adding a mark box besides minor and watch options would be great (for bots, admins and flooders) I think so too. Display it only to people with the bot flag, and their edits will be tagged as bot edits iff they check the box. The API would be unaffected, and continue to function as it did before. As a product manager I am wary of adding more clutter into the already busy edit interface, but given the scope of the feature (i.e. it only displays to people flagged as bots), I think it'd be okay. Dan -- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
On May 20, 2014 8:39 AM, Dan Garry dga...@wikimedia.org wrote:
On 19 May 2014 19:36, Amir Ladsgroup ladsgr...@gmail.com wrote:
As a bot operator I think API parameter about flagging bot or not is
necessary
Sure, but as I'm not a bot operator, can you explain why and what you use
this for, to help me understand? :-)
Implementing the parameter (whether the current status quo or my
suggestion) in UI is important, I think adding a mark box besides minor
and
watch options would be great (for bots, admins and flooders)
I think so too. Display it only to people with the bot flag, and their
edits will be tagged as bot edits iff they check the box. The API would be
unaffected, and continue to function as it did before.
As a product manager I am wary of adding more clutter into the already
busy
edit interface, but given the scope of the feature (i.e. it only displays
to people flagged as bots), I think it'd be okay.
If it is role based ('bot'), why not change it (in 1.24?) to always *not*
flag the UI edits as bot edits, displaying a warning to that effect?
Do we have many UI scaping bot frameworks in use these days?
--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
Because humans use it these days, not boys generally in the web interface
and it would just make stuff harder for people that use it…
On Tuesday, May 20, 2014, John Mark Vandenberg jay...@gmail.com wrote:
On May 20, 2014 8:39 AM, Dan Garry dga...@wikimedia.org javascript:;
wrote:
On 19 May 2014 19:36, Amir Ladsgroup ladsgr...@gmail.com javascript:;
wrote:
As a bot operator I think API parameter about flagging bot or not is
necessary
Sure, but as I'm not a bot operator, can you explain why and what you use
this for, to help me understand? :-)
Implementing the parameter (whether the current status quo or my
suggestion) in UI is important, I think adding a mark box besides minor
and
watch options would be great (for bots, admins and flooders)
I think so too. Display it only to people with the bot flag, and their
edits will be tagged as bot edits iff they check the box. The API would
be
unaffected, and continue to function as it did before.
As a product manager I am wary of adding more clutter into the already
busy
edit interface, but given the scope of the feature (i.e. it only displays
to people flagged as bots), I think it'd be okay.
If it is role based ('bot'), why not change it (in 1.24?) to always *not*
flag the UI edits as bot edits, displaying a warning to that effect?
Do we have many UI scaping bot frameworks in use these days?
--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org javascript:;
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Sent from Gmail Mobile on my iPod.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Bot flags and human-made edits
On Tue, May 20, 2014 at 3:39 AM, Dan Garry dga...@wikimedia.org wrote: On 19 May 2014 19:36, Amir Ladsgroup ladsgr...@gmail.com wrote: As a bot operator I think API parameter about flagging bot or not is necessary Sure, but as I'm not a bot operator, can you explain why and what you use this for, to help me understand? :-) I think an example of bot using this API parameter is Salebot on frwiki : when it reverts a vandalism, this edit is not marked with the bot flag so that it appears in the recent changes, and made human aware that a vandalism has been reverted. For other things, it may have its edit marked with the bot flag. Nico ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Student software dev projects for Wikimedia
Hi, a few of us had an offline discussion about getting university professors to encourage their students to publish code that they develop for classes based on MediaWiki and Wikimedia projects. What I heard is that professors are using MW and Wikimedia projects as environments for student devs but not many are publishing the code that the students produce. Is anyone working on outreach to university professors to encourage them to get student projects published and in a form that we can use, and is there a list of projects somewhere that are suggested for professors to use when teaching? This could be a dev equivalent to the Wikimedia Education Program. Pine ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
