Re: [Wikitech-l] [Tech Talks] June 25, 2019, 6 PM UTC, Just what is Analytics doing back there?

[Nuria Ruiz]

The talk has started, this is the you tube stream:
https://www.youtube.com/watch?reload=9=GD0PEDFysfM


Thanks,

Nuria

On Mon, Jun 10, 2019 at 8:53 AM Subramanya Sastry 
wrote:

> Hi Everyone,
>
> It's time for Wikimedia Tech Talks 2019 Episode 5!
> This month's talk will take place *June 25, 2019 at 6:00 PM UTC.*
>
> *Topic*: Just what is Analytics doing back there?
>
> *Speaker*: Dan Andreescu, Senior Software Engineer, Analytics
>
> *Summary*: We take care of twelve systems. Data flows through them to
> answer the many questions that our community and staff have about
> our piece of the open knowledge movement. Let's take a look at how
> these systems fit together and dive deeper into some of the more
> interesting algorithms.
>
> *YouTube stream for viewers*: https://www.youtube.com/watch?v=GD0PEDFysfM
>
> During the live talk, you are invited to join the discussion on IRC
> at #wikimedia-office
>
> You can watch past Tech Talks here:
> https://www.mediawiki.org/wiki/Tech_talks
>
> If you are interested in giving your own tech talk, you can learn more
> here:
>
> https://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event#Tech_talks
>
>
> Subbu.
> (Standing in for Sarah Rodlund who is currently away)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] New time selector in wikistats2 UI

[Nuria Ruiz]

Hello,

Over the last couple months we've been working on improving the experience
of looking through the past on Wikistats2. Until now simple questions like
"who were the top editors in June 2010" or "what countries were visiting
Arabic Wikipedia the most in 2004" were difficult to answer because of our
very limited time selection options on the UI.

This week we deployed the *new time range selector* on Wikistats. As
opposed to the old one, it works for both top and time-series metrics. It
can be used to share links to specific periods in any metric of any wiki.
And we've added a toggle button to switch between monthly and daily
granularities. Please check it out, see, for example, edits for Italian
wikipedia in content spaces between December 2014 and April 2019:

https://stats.wikimedia.org/v2/#/it.wikipedia.org/contributing/edits/normal|bar|2014-12-01~2019-04-30|page_type~content|monthly


As always feedback welcome, handy link to report bugs:
https://phabricator.wikimedia.org/maniphest/task/edit/?title=Wikistats%20Bug=Analytics-Wikistats,Analytics

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Easier mapping from Wikistats1 to Wikistats2 metrics

[Nuria Ruiz]

Hello!

Analytics team would like to announce couple changes. We are working
towards an easier way to navigate metrics that appear in both Wikistats1
and Wikistats2 and compare numbers, please take a look at changes deployed
today for (for example) Italian Wikipedia:
https://stats.wikimedia.org/v2/#/metrics/it.wikipedia.org

This is an example of a definition of active editors that matches
Wikistats1:

https://stats.wikimedia.org/v2/#/it.wikipedia.org/contributing/active-editors/normal|line|2-Year|~total

As always please file bugs (suggestions are fine too) on Phab using this
handy link:

https://phabricator.wikimedia.org/maniphest/task/edit/?title=Wikistats%20Bug=Analytics-Wikistats,Analytics

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] A difficult goodbye

[Nuria Ruiz]

Many thanks for your work in these two years, Victoria.  You leave the
Technology team in much better shape than you found it.

For those of you not in the know I think is worth mentioning that in her
tenure here Victoria has created the Technical Engagement team to better
attend technical contributors (already mentioned on this thread), hired a
director of security that has, in turn, build a solid security team, given
weight to MediaWiki as a platform and created a team that will steward
MediaWiki's development . She has also overseen much of the technical work
that has gone into our litigation against the NSA, has instituted a process
to better do promotions and has hired support staff that has made the work
of all of us in the department much (much!) easier.

On Mon, Jan 14, 2019 at 12:02 PM Nuria Ruiz  wrote:

> Many thanks for your work in these two years, Victoria.  You leave the
> Technology team in much better shape than you found it.
>
> For those of you not in the know I think is worth mentioning that in her
> tenure here Victoria has created the Technical Engagement team to better
> attend technical contributors (already mentioned on this thread), hired a
> director of security that has, in turn, build a solid security team, given
> weight to MediaWiki as a platform and created a team that will steward
> MediaWiki's development . She has also overseen much of the technical work
> that has gone into our litigation against the NSA, has instituted a process
> to better do promotions and has hired support staff that has made the work
> of all of us in the department much (much!) easier.
>
>
>
> On Mon, Jan 14, 2019 at 10:17 AM Chico Venancio 
> wrote:
>
>> Victoria,
>>
>> I echo Isarra's comment here, it is the Wikimedia community loss to see
>> you
>> part.
>> In particular, I was able to see your leadership in the formation of the
>> Technical Engagement team, consolidating initiatives around volunteer
>> developers and leveraging the work for the communities.
>>
>> Wish you the best on this new challenge.
>> Chico Venancio
>>
>>
>> Em seg, 14 de jan de 2019 às 15:06, Isarra Yos 
>> escreveu:
>>
>> > Sad to see you go - from what I saw as a volunteer, things only improved
>> > under your leadership here. We were lucky to have you.
>> >
>> > All the best.
>> >
>> > -I
>> >
>> > On 11/01/2019 00:29, Victoria Coleman wrote:
>> > > Dear all,
>> > >
>> > > I have some difficult news to share.
>> > >
>> > > A few months ago and completely out of the blue, an opportunity came
>> up
>> > for me to exercise the full spectrum of my skills as the CEO of an early
>> > stage mission oriented startup. It has been a mighty struggle between my
>> > commitment to my team, the Foundation and  the movement and this
>> > opportunity to bring all my skills to bear and build something from the
>> > ground up to do good in the world. I will not lie to you - it has not
>> been
>> > easy. But ultimately I decided that I have to give it a go and I
>> accepted
>> > the offer. I plan on starting there on Feb 4th so my last day at the
>> > Foundation will be Feb 1st.
>> > >
>> > > These past two years have been amongst the most enjoyable in my
>> > professional career. I’ve enjoyed getting to know the movement and what
>> > fuels it and I’ve been incredibly privileged to work with a Tech team
>> like
>> > no other. We have together strengthened the technical infrastructure of
>> the
>> > movement and while much work remains to be done we have a much stronger
>> > team in place to take the mission to the next level. And I have
>> personally
>> > made friendships that will last a lifetime.
>> > >
>> > > The organization I will be moving to is Atlas AI, a public benefit
>> > corporation supported by the Rockefeller Foundation. I will be leading
>> an
>> > exceptional team of AI and earth system science experts striving for
>> > improvements in human well being through data driven insights. Our focus
>> > areas are drawn from the UN’s sustainable development goals of no
>> poverty
>> > and zero hunger with emphasis on Sub Saharan Africa.
>> > >
>> > > I leave behind a Technology department that I am certain is well on
>> the
>> > way to achieving our vision to create the infrastructure for the free
>> > knowledge movement. I am also pleased that during my tenure, we have
>> built
>> > leaders who are equipped to take on this ch

Re: [Wikitech-l] A difficult goodbye

[Nuria Ruiz]

Many thanks for your work in these two years, Victoria.  You leave the
Technology team in much better shape than you found it.

For those of you not in the know I think is worth mentioning that in her
tenure here Victoria has created the Technical Engagement team to better
attend technical contributors (already mentioned on this thread), hired a
director of security that has, in turn, build a solid security team, given
weight to MediaWiki as a platform and created a team that will steward
MediaWiki's development . She has also overseen much of the technical work
that has gone into our litigation against the NSA, has instituted a process
to better do promotions and has hired support staff that has made the work
of all of us in the department much (much!) easier.



On Mon, Jan 14, 2019 at 10:17 AM Chico Venancio 
wrote:

> Victoria,
>
> I echo Isarra's comment here, it is the Wikimedia community loss to see you
> part.
> In particular, I was able to see your leadership in the formation of the
> Technical Engagement team, consolidating initiatives around volunteer
> developers and leveraging the work for the communities.
>
> Wish you the best on this new challenge.
> Chico Venancio
>
>
> Em seg, 14 de jan de 2019 às 15:06, Isarra Yos 
> escreveu:
>
> > Sad to see you go - from what I saw as a volunteer, things only improved
> > under your leadership here. We were lucky to have you.
> >
> > All the best.
> >
> > -I
> >
> > On 11/01/2019 00:29, Victoria Coleman wrote:
> > > Dear all,
> > >
> > > I have some difficult news to share.
> > >
> > > A few months ago and completely out of the blue, an opportunity came up
> > for me to exercise the full spectrum of my skills as the CEO of an early
> > stage mission oriented startup. It has been a mighty struggle between my
> > commitment to my team, the Foundation and  the movement and this
> > opportunity to bring all my skills to bear and build something from the
> > ground up to do good in the world. I will not lie to you - it has not
> been
> > easy. But ultimately I decided that I have to give it a go and I accepted
> > the offer. I plan on starting there on Feb 4th so my last day at the
> > Foundation will be Feb 1st.
> > >
> > > These past two years have been amongst the most enjoyable in my
> > professional career. I’ve enjoyed getting to know the movement and what
> > fuels it and I’ve been incredibly privileged to work with a Tech team
> like
> > no other. We have together strengthened the technical infrastructure of
> the
> > movement and while much work remains to be done we have a much stronger
> > team in place to take the mission to the next level. And I have
> personally
> > made friendships that will last a lifetime.
> > >
> > > The organization I will be moving to is Atlas AI, a public benefit
> > corporation supported by the Rockefeller Foundation. I will be leading an
> > exceptional team of AI and earth system science experts striving for
> > improvements in human well being through data driven insights. Our focus
> > areas are drawn from the UN’s sustainable development goals of no poverty
> > and zero hunger with emphasis on Sub Saharan Africa.
> > >
> > > I leave behind a Technology department that I am certain is well on the
> > way to achieving our vision to create the infrastructure for the free
> > knowledge movement. I am also pleased that during my tenure, we have
> built
> > leaders who are equipped to take on this challenge.
> > >
> > > Erika Bjune will be serving as interim CTO. Erika joined us a little
> > over two years ago and she has distinguished  herself as one of the
> finest
> > people leaders I have ever worked with. Both she and Katherine will have
> my
> > ongoing support — I will stay on in a consulting role to Erika on
> > organizational matters and for the mid term planning work we have ahead
> of
> > us in the remainder of the fiscal year.  I know that I am leaving the
> Tech
> > team in good hands until a permanent CTO is hired.
> > >
> > > I want to close this difficult message with my heartfelt thanks to all
> > of you for letting me be part of this incredible movement. Being a
> > Wikimedian is a great privilege. I wanted you to know that I’m not
> walking
> > away from something; I am walking towards  something that is very
> > important to me and the world. I will miss you all. Deeply. Truly. And a
> > lot!
> > >
> > >
> > > Victoria
> > >
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___

[Wikitech-l] Wikistats2 - Metrics available for project families

[Nuria Ruiz]

Hello!

The Analytics team would like to announce that we have now in Wikistats2
metrics available for what we are calling (for the lack of a better name)
"project families". That is, "all wikipedias", "all wikibooks"..etc

See, for example, bytes added by users to all wikibooks in the last month:
https://stats.wikimedia.org/v2/#/all-wikibooks-projects/content/net-bytes-difference/normal|bar|1-Month|editor_type~user

And "all wikibooks top editors" [2]:

https://stats.wikimedia.org/v2/#/all-wikibooks-projects/contributing/top-editors/normal|table|1-Month|editor_type~user

Not all metrics are available per project, most notably we (yet) do not
have pageviews. As always please file bugs [2] if you find any, and let us
know what can we do better.

Thanks,

Nuria

[1] https://meta.wikimedia.org/wiki/Research:Wikistats_metrics/Top_editors
[2]
https://phabricator.wikimedia.org/maniphest/task/edit/?title=Wikistats%20Bug=Analytics-Wikistats,Analytics
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] New Reports in wikistats2: "top editors" (a.k.a most prolific contributors) and "top edited articles"

[Nuria Ruiz]

Hello,

The analytics team would like to announce two new metrics available in
wikistats2:

1. Top editors (a.k.a most prolific contributors)
See example for Italian wikipedia:

https://stats.wikimedia.org/v2/#/it.wikipedia.org/contributing/top-editors/normal|table|1-Month|~total

2. Top edited articles (pages with most edits, not most contributors):
Again, example for Italian wikipedia:

https://stats.wikimedia.org/v2/#/it.wikipedia.org/contributing/top-edited-pages/normal|table|1-Month|~total

Please take a look and, as always, send feedback via phab or irc
(#wikimedia-analytics)

The tasks we have in our radar for wikistats2 are metrics "per family",
that is. "edits for all wikitionary projects"  or "unique devices for all
wikipedias".

Some of these metrics are already available in the API. See for example,
the daily number of edits for all wikitionary.org projects for August 2018,
made by registered users on articles (pages in content namespace):

https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/all-wiktionary-projects/user/content/daily/20180801/20180901

More info about edit data apis here:
https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats_2

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] My Phabricator account has been disabled

[Nuria Ruiz]

...
> >>
> >> On Tue, Aug 14, 2018 at 4:08 PM, David Barratt 
> >> wrote:
> >> >>
> >> >> Again, this isn't enwiki, but there would be a large mob gathering at
> >> the
> >> >> administrators' doorstep on enwiki for a block without that context
> and
> >> >> backstory.
> >> >>
> >> >
> >> > That seems like really toxic behavior.
> >> >
> >> > On Tue, Aug 14, 2018 at 6:27 AM George Herbert <
> george.herb...@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> I keep seeing "abusers" and I still haven't seen the evidence of the
> >> >> alleged long term abuse pattern.
> >> >>
> >> >> Again, this isn't enwiki, but there would be a large mob gathering at
> >> the
> >> >> administrators' doorstep on enwiki for a block without that context
> and
> >> >> backstory.  That's not exactly the standard here, but ... would
> someone
> >> >> just answer the question?  What happened leading up to this to
> justify
> >> the
> >> >> block?  If it's that well known, you can document it.
> >> >>
> >> >> On Tue, Aug 14, 2018 at 12:18 AM, Adam Wight 
> >> wrote:
> >> >>
> >> >> > Hi Petr,
> >> >> >
> >> >> > Nobody is language policing, this is about preventing abusive
> behavior
> >> >> and
> >> >> > creating an inviting environment where volunteers and staff don't
> >> have to
> >> >> > waste time with emotional processing of traumatic interactions.
> >> >> >
> >> >> > I think we're after the same thing, that we want to keep our
> community
> >> >> > friendly and productive, so it's just a matter of agreeing on the
> >> means
> >> >> to
> >> >> > accomplish this.  I see the Code of Conduct Committee standing up
> to
> >> the
> >> >> > nonsense and you see them as being hostile, so our perspectives
> >> diverge
> >> >> at
> >> >> > that point.  I also see lots of people on this list standing up for
> >> what
> >> >> > they think is right, and I'd love if that energy could be organized
> >> >> better
> >> >> > so that we're not sniping at each other, but instead refining our
> >> shared
> >> >> > statements of social values and finding a way to encourage the good
> >> while
> >> >> > more effectively addressing the worst in us.
> >> >> >
> >> >> > This isn't coherent enough to share yet, but I'll try anyway—I've
> been
> >> >> > thinking about how our high proportion of anarchic- and
> >> >> > libertarian-oriented individuals helped shape a culture which
> doesn't
> >> >> > handle "negative laws" [1] well.  For example, the Code of Conduct
> is
> >> >> > mostly focused on "unacceptable behaviors", but perhaps we could
> >> rewrite
> >> >> it
> >> >> > in the positive sense, as a set of shared responsibilities to
> support
> >> >> each
> >> >> > other and the less powerful person in any conflict.  We have a duty
> to
> >> >> > speak up, a duty to keep abusers from their target, we own this
> social
> >> >> > space and have to maintain it together.  If you see where I'm
> headed?
> >> >> > Rewriting the CoC in a positive rights framework is a daunting
> >> project,
> >> >> but
> >> >> > it might be fun.
> >> >> >
> >> >> > Regards,
> >> >> > Adam
> >> >> >
> >> >> > [1] https://en.wikipedia.org/wiki/Negative_and_positive_rights
> >> >> >
> >> >> > On Mon, Aug 13, 2018 at 9:36 AM Petr Bena 
> wrote:
> >> >> >
> >> >> > > I am a bit late to the party, but do we seriously spend days
> >> >> > > discussing someone being banned from a bug tracker just for
> saying
> >> >> > > "WTF", having their original comment completely censored, so that
> >> the
> >> >> > > community can't even make a decision how bad it really was? Is
> that
> >> >> > > what we turned into?

Re: [Wikitech-l] My Phabricator account has been disabled

[Nuria Ruiz]

>After several negative examples discussed in the last few months on this
list,* this action conclusively proves in my eyes the failure of the Code
of conduct to be a positive force for our community, at least so far >and
in the present conditions.
The CoC will prioritize the safety of the minority over the comfort of the
majority. It might not be perfect but it is sure already working well to
echo and document the concerns of many in the community that really do not
feel comfortable to reply to a thread in this e-mail list. Or phabricator.
Or a talk page. Reports are confidential and would continue to be so to
make sure everyone feels safe to report *any* incident of *any* severity.
On this specific case discussing whether profanity is OK or not is just a
distraction as there is a long history of hostility and harsh criticism.
The way the bug was closed might be incorrect (I personally as an engineer
agree that closing it shows little understanding of how technical teams do
track bugs in phab, some improvements are in order here for sure) but the
harsh interaction is just one out of many that have been out of line for
while.

Thanks,

Nuria



On Wed, Aug 8, 2018 at 8:42 PM, Federico Leva (Nemo) 
wrote:

> Thanks Amir and MZMcBride for disclosing the action.
>
> A volunteer has been punished for speaking up in defense of fellow
> volunteer and paid contributors, whose contribution was being sidelined and
> suffocated by people "in charge" of the specific space, i.e. the people
> they were doing their best to help.
>
> The message which was sanctioned was even of an especially thoughtful
> kind, in my opinion, because it didn't attempt to submerge the other users
> with walls of text, politically correct tirades or otherwise charged
> statements. It was merely a heartfelt interjection to help people stop,
> reconsider their actions and self-improve without the need of lectures. Was
> this peculiar effort at constructive facilitation considered? If not, what
> alternatives or constructive suggestions were provided?
>
> After several negative examples discussed in the last few months on this
> list,* this action conclusively proves in my eyes the failure of the Code
> of conduct to be a positive force for our community, at least sso far and
> in the present conditions.
>
> The committee needs to immediately resign or be disbanded, and be reformed
> on a more solid basis.
>
> Federico
>
> (*) And not a single disclosed positive action, as far as I know. But it
> might have been lost due to the lack of a transparency report. If one was
> released or is upcoming, sorry; I'll revise my conclusions accordingly.
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Better maps in Wikistats2 and new metric: Legacy Pageviews (a.k.a Pagecounts)

[Nuria Ruiz]

On Wed, Jul 11, 2018 at 1:26 PM, Nuria Ruiz  wrote:

> Hello!
>
> Just  a brief note to announce that we have two new things in wikistats
> this quarter. We have reviewed maps by popular demand to give more precise
> pageviews per country.
>
> Check, for example, pageviews for portuguese wikipedia on the world for
> last month:
>
> https://stats.wikimedia.org/v2/#/pt.wikipedia.org/reading/
> page-views-by-country/normal|map|2-Year~2016060100~2018071100|~total
>
> Also, we have included legacy pageviews to the UI, we used to call these
> page-counts and prior to June 2015 this is the metric that we reported as
> pageviews for all wikimedia sites.
>
> Info about metric:
> https://wikitech.wikimedia.org/wiki/Analytics/Archive/Data/Pagecounts-raw
>
> See, for example, pagecounts for portuguese wikipedia from 2008 to 2016.
> https://stats.wikimedia.org/v2/#/pt.wikipedia.org/reading/
> legacy-page-views/normal|bar|All~1980010100~2018071100|~total
>
> Also, all urls are now bookmarkable.
>
> As always suggestions welcome, please file bug reports on phabricator.
>
> Thanks,
>
> Nuria
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Batter maps in Wikistats2 and new metric: Legacy Pageviews (a.k.a Pagecounts)

[Nuria Ruiz]

Hello!

Just  a brief note to announce that we have two new things in wikistats
this quarter. We have reviewed maps by popular demand to give more precise
pageviews per country.

Check, for example, pageviews for portuguese wikipedia on the world for
last month:

https://stats.wikimedia.org/v2/#/pt.wikipedia.org/reading/page-views-by-country/normal|map|2-Year~2016060100~2018071100|~total

Also, we have included legacy pageviews to the UI, we used to call these
page-counts and prior to June 2015 this is the metric that we reported as
pageviews for all wikimedia sites.

Info about metric:
https://wikitech.wikimedia.org/wiki/Analytics/Archive/Data/Pagecounts-raw

See, for example, pagecounts for portuguese wikipedia from 2008 to 2016.
https://stats.wikimedia.org/v2/#/pt.wikipedia.org/reading/legacy-page-views/normal|bar|All~1980010100~2018071100|~total

Also, all urls are now bookmarkable.

As always suggestions welcome, please file bug reports on phabricator.

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

> I think you want to set zoom limits - right now you can zoom in and zoom out
almost infinitely.
This issue should be fixed now. Please see:
https://stats.wikimedia.org/v2/#/fr.wikipedia.org/reading/page-views-by-country


On Wed, Feb 14, 2018 at 6:24 PM, Yuri Astrakhan <yuriastrak...@gmail.com>
wrote:

> Nuria, well done, looks awesome!  I think you want to set zoom limits -
> right now you can zoom in and zoom out almost infinitely. Congrats!
>
> On Wed, Feb 14, 2018 at 8:47 PM, Michael Schönitzer <
> michael.schoenit...@wikimedia.de> wrote:
>
> > here a screenshot:
> >
> > There's a second anomaly: "SX" is also listed…
> >
> > 2018-02-15 1:35 GMT+01:00 יגאל חיטרון <khit...@gmail.com>:
> >
> > > Sorry, I can't. I opened the link you gave on hewiki. Changed from map
> > view
> > > to table view. I get a list of countries - US, France, Spain, --,
> Japan,
> > > and so on. It's a link, and clicking it opens unexisting wiki article
> > with
> > > the same name.
> > > Igal
> > >
> > >
> > > On Feb 15, 2018 02:23, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > >Hello and thank you.
> > > > >What do you mean in country named "--"?
> > > >
> > > > Not sure what you are asking, maybe a screenshot would help?
> > > >
> > > > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com>
> > wrote:
> > > >
> > > > > Hello and thank you.
> > > > > What do you mean in country named "--"?
> > > > > Igal (User:IKhitron)
> > > > >
> > > > >
> > > > > On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
> > > > >
> > > > > Hello from Analytics team:
> > > > >
> > > > > Just a brief note to announce that Wikistats 2.0 includes data
> about
> > > > > pageviews per project per country for the current month.
> > > > >
> > > > > Take a look, pageviews for Spanish Wikipedia this current month:
> > > > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> > > > > pageviews-by-country
> > > > >
> > > > > Data is also available programatically vi APIs:
> > > > >
> > > > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> > > > > Pageviews#Pageviews_split_by_country
> > > > >
> > > > > We will be deploying small UI tweaks during this week but please
> > > explore
> > > > > and let us know what you think.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Nuria
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> >
> >
> >
> > --
> > Michael F. Schönitzer
> >
> >
> >
> > Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> > Tel. (030) 219 158 26-0
> > http://wikimedia.de
> >
> > Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> > Wissens frei teilhaben kann. Helfen Sie uns dabei!
> > http://spenden.wikimedia.de/
> >
> > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
> > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter
> > der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> > Körperschaften I Berlin, Steuernummer 27/681/51985.
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Better Support for Mobile in Wikistats2

[Nuria Ruiz]

Hello!

Analytics is working on better support for mobile in wikistats2:
http://stats.wikimedia.org/v2

Do take a look at latest changes and if there are issues that prevent you
from using this site in mobile let us know.

A phabricator ticket (http://phabricator.wikimedia.org) with a screenshot
will be great.

Site should be usable in:
* last two versions of any major desktop browser (chrome, FF, edge and
safari)
* any newish iOS device (iphone4 and up) that has been kept up to date
* android 4.4 and up

We are still working on improving our mobile support and happy to take
requests on higher levels of support for browsers we might have not
considered.

This is the work we have planned for next quarter:

   - Persistent links for metrics. Bookmarks. task T179444
   
   - Support annotations task T178015
   
   - Label non obvious concepts task T177950
   
   - Improvements on pageview data per country task T188928
   


Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] You can now translate Phabricator to your language

[Nuria Ruiz]

What an awesome project translatewiki is.

Nuria

On Fri, Mar 2, 2018 at 11:59 AM, Daniel Zahn  wrote:

> Great work! It made me create a fresh TranslateWiki user and add some
> German translations of Phabricator strings.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>It took me a while to understand the meaning of 100M->1G and friends.
Indeed, from other users we also gathered it was pretty confusing.  We have
changed that and while localizing the site is something we have planned for
later this year we have gone the way of localizing number formatting and
units according to your browser language (for the locales supported by
numeral.js [1]).

See how wikistats looks if your browser language is Japanese [2], Italian:
[3] or Russian [4]

This is not a common choice when it comes to localization but one that
might fit better our userbase, we'll see.

[1] http://numeraljs.com/ Pull requests for locales: https://github.com/
adamwdraper/Numeral-js/tree/master/locales

[2] https://imgur.com/a/sqHMZ
[3] https://imgur.com/a/1FsBE
[4] https://imgur.com/a/PBMrY


On Thu, Feb 15, 2018 at 8:33 AM, Nuria Ruiz <nu...@wikimedia.org> wrote:

>
> >Nuria, well done, looks awesome!  I think you want to set zoom limits -
> >right now you can zoom in and zoom out almost infinitely.
> Fix on the way:
> https://phabricator.wikimedia.org/T187205
>
> On Wed, Feb 14, 2018 at 6:24 PM, Yuri Astrakhan <yuriastrak...@gmail.com>
> wrote:
>
>> Nuria, well done, looks awesome!  I think you want to set zoom limits -
>> right now you can zoom in and zoom out almost infinitely. Congrats!
>>
>> On Wed, Feb 14, 2018 at 8:47 PM, Michael Schönitzer <
>> michael.schoenit...@wikimedia.de> wrote:
>>
>> > here a screenshot:
>> >
>> > There's a second anomaly: "SX" is also listed…
>> >
>> > 2018-02-15 1:35 GMT+01:00 יגאל חיטרון <khit...@gmail.com>:
>> >
>> > > Sorry, I can't. I opened the link you gave on hewiki. Changed from map
>> > view
>> > > to table view. I get a list of countries - US, France, Spain, --,
>> Japan,
>> > > and so on. It's a link, and clicking it opens unexisting wiki article
>> > with
>> > > the same name.
>> > > Igal
>> > >
>> > >
>> > > On Feb 15, 2018 02:23, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > >Hello and thank you.
>> > > > >What do you mean in country named "--"?
>> > > >
>> > > > Not sure what you are asking, maybe a screenshot would help?
>> > > >
>> > > > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Hello and thank you.
>> > > > > What do you mean in country named "--"?
>> > > > > Igal (User:IKhitron)
>> > > > >
>> > > > >
>> > > > > On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>> > > > >
>> > > > > Hello from Analytics team:
>> > > > >
>> > > > > Just a brief note to announce that Wikistats 2.0 includes data
>> about
>> > > > > pageviews per project per country for the current month.
>> > > > >
>> > > > > Take a look, pageviews for Spanish Wikipedia this current month:
>> > > > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
>> > > > > pageviews-by-country
>> > > > >
>> > > > > Data is also available programatically vi APIs:
>> > > > >
>> > > > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
>> > > > > Pageviews#Pageviews_split_by_country
>> > > > >
>> > > > > We will be deploying small UI tweaks during this week but please
>> > > explore
>> > > > > and let us know what you think.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Nuria
>> > > > > ___
>> > > > > Wikitech-l mailing list
>> > > > > Wikitech-l@lists.wikimedia.org
>> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> > > > > ___
>> > > > > Wikitech-l mailing list
>> > > > > Wikitech-l@lists.wikimedia.org
>> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> > > > ___
>> > > > Wikitech-l mailing list
>> > > > Wikitech-l@lists.wikimedia.org
>

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>How is this new metric going to differ from that?
Differ? In no way, the ticket [1] is about computing the value not
re-defining it.
Wikistats2 only computes (at this time) active editors (per wikistats
definition) per project.


[1] https://phabricator.wikimedia.org/T188265


On Wed, Feb 28, 2018 at 1:30 PM, Tilman Bayer <tba...@wikimedia.org> wrote:

> The number of active editors for all projects is a core metric that WMF has
> tracked and reported on a regular basis since at least 2011 (with the
> definition having been revised  in the last year or two).
>
> https://www.mediawiki.org/wiki/Wikimedia_Audiences#Contributors
> https://stats.wikimedia.org/reportcard/RC_2011_12_detailed.html
> https://meta.wikimedia.org/wiki/Research:Active_editor
> https://meta.wikimedia.org/wiki/Research:Defining_
> monthly_active_editors,_2016
>
> How is this new metric going to differ from that? Or are we talking about
> the same thing?
>
> On Mon, Feb 26, 2018 at 8:42 AM, Nuria Ruiz <nu...@wikimedia.org> wrote:
>
> > Created a ticket to compute "active editors for all wikis":
> > https://phabricator.wikimedia.org/T188265
> >
> > On Sat, Feb 24, 2018 at 3:56 PM, Nuria Ruiz <nu...@wikimedia.org> wrote:
> >
> > > >By the way, is there somewhere where I could find total active editors
> > > of all wikisources?
> > >
> > > All projects agreggated? Not as far as I know. More than a matter of
> > > calculation I think is a matter of definition. I might be wrong though
> > and
> > > this might already have been discussed.
> > >
> > > Active editor metric is defined "per wiki" to my knowledge:
> https://meta
> > .
> > > wikimedia.org/wiki/Research:Wikistats_metrics/Editors
> > >
> > > For all projects you could add "all active editors for all projects"
> and
> > > that is doable. Now,  in that case you will not count someone with 2
> > edits
> > > in eswiki and (using same login) another 3 edits on dewiki if in order
> to
> > > become an "active editor" you need 5 edits in one wiki.
> > >
> > > Phab ticket filed: https://phabricator.wikimedia.org/T188194
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Feb 22, 2018 at 1:37 AM, mathieu stumpf guntz <
> > > psychosl...@culture-libre.org> wrote:
> > >
> > >> Hi Nuria,
> > >>
> > >> Thank you for the report, and congratulation to all people involved in
> > >> releasing this tool. It would be fine to also have map with editiors
> and
> > >> active editors.
> > >>
> > >> By the way, is there somewhere where I could find total active editors
> > of
> > >> all wikisources? Surely I could sum that through API (providing that
> it
> > >> exposes such a data for each language), but it would be fine that
> > everybody
> > >> could have a straight forward access to this kind of cross language
> > data. I
> > >> think there real are usecases for that, actually I'm looking for
> number
> > of
> > >> active wikisourcerer to evaluate number of attendees we might set for
> > the
> > >> next Wikisource conference.
> > >>
> > >> Cheers.
> > >>
> > >> Le 14/02/2018 à 23:15, Nuria Ruiz a écrit :
> > >>
> > >> Hello from Analytics team:
> > >>
> > >> Just a brief note to announce that Wikistats 2.0 includes data about
> > >> pageviews per project per country for the current month.
> > >>
> > >> Take a look, pageviews for Spanish Wikipedia this current month:
> > https://stats.wikimedia.org/v2/#/es.wikipedia.org/read
> > ing/pageviews-by-country
> > >>
> > >> Data is also available programatically vi APIs:
> > >> https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#
> > Pageviews_split_by_country
> > >>
> > >>
> > >> We will be deploying small UI tweaks during this week but please
> explore
> > >> and let us know what you think.
> > >>
> > >> Thanks,
> > >>
> > >> Nuria
> > >> ___
> > >> Wikitech-l mailing listWikitech-l@lists.wikimedia.orghttps://
> > lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >>
> > >>
> > >>
> > >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
>
>
>
> --
> Tilman Bayer
> Senior Analyst
> Wikimedia Foundation
> IRC (Freenode): HaeB
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

Created a ticket to compute "active editors for all wikis":
https://phabricator.wikimedia.org/T188265

On Sat, Feb 24, 2018 at 3:56 PM, Nuria Ruiz <nu...@wikimedia.org> wrote:

> >By the way, is there somewhere where I could find total active editors
> of all wikisources?
>
> All projects agreggated? Not as far as I know. More than a matter of
> calculation I think is a matter of definition. I might be wrong though and
> this might already have been discussed.
>
> Active editor metric is defined "per wiki" to my knowledge: https://meta.
> wikimedia.org/wiki/Research:Wikistats_metrics/Editors
>
> For all projects you could add "all active editors for all projects"  and
> that is doable. Now,  in that case you will not count someone with 2 edits
> in eswiki and (using same login) another 3 edits on dewiki if in order to
> become an "active editor" you need 5 edits in one wiki.
>
> Phab ticket filed: https://phabricator.wikimedia.org/T188194
>
>
>
>
>
>
> On Thu, Feb 22, 2018 at 1:37 AM, mathieu stumpf guntz <
> psychosl...@culture-libre.org> wrote:
>
>> Hi Nuria,
>>
>> Thank you for the report, and congratulation to all people involved in
>> releasing this tool. It would be fine to also have map with editiors and
>> active editors.
>>
>> By the way, is there somewhere where I could find total active editors of
>> all wikisources? Surely I could sum that through API (providing that it
>> exposes such a data for each language), but it would be fine that everybody
>> could have a straight forward access to this kind of cross language data. I
>> think there real are usecases for that, actually I'm looking for number of
>> active wikisourcerer to evaluate number of attendees we might set for the
>> next Wikisource conference.
>>
>> Cheers.
>>
>> Le 14/02/2018 à 23:15, Nuria Ruiz a écrit :
>>
>> Hello from Analytics team:
>>
>> Just a brief note to announce that Wikistats 2.0 includes data about
>> pageviews per project per country for the current month.
>>
>> Take a look, pageviews for Spanish Wikipedia this current 
>> month:https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/pageviews-by-country
>>
>> Data is also available programatically vi APIs:
>> https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Pageviews_split_by_country
>>
>>
>> We will be deploying small UI tweaks during this week but please explore
>> and let us know what you think.
>>
>> Thanks,
>>
>> Nuria
>> ___
>> Wikitech-l mailing 
>> listWikitech-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>By the way, is there somewhere where I could find total active editors of
all wikisources?

All projects agreggated? Not as far as I know. More than a matter of
calculation I think is a matter of definition. I might be wrong though and
this might already have been discussed.

Active editor metric is defined "per wiki" to my knowledge:
https://meta.wikimedia.org/wiki/Research:Wikistats_metrics/Editors

For all projects you could add "all active editors for all projects"  and
that is doable. Now,  in that case you will not count someone with 2 edits
in eswiki and (using same login) another 3 edits on dewiki if in order to
become an "active editor" you need 5 edits in one wiki.

Phab ticket filed: https://phabricator.wikimedia.org/T188194






On Thu, Feb 22, 2018 at 1:37 AM, mathieu stumpf guntz <
psychosl...@culture-libre.org> wrote:

> Hi Nuria,
>
> Thank you for the report, and congratulation to all people involved in
> releasing this tool. It would be fine to also have map with editiors and
> active editors.
>
> By the way, is there somewhere where I could find total active editors of
> all wikisources? Surely I could sum that through API (providing that it
> exposes such a data for each language), but it would be fine that everybody
> could have a straight forward access to this kind of cross language data. I
> think there real are usecases for that, actually I'm looking for number of
> active wikisourcerer to evaluate number of attendees we might set for the
> next Wikisource conference.
>
> Cheers.
>
> Le 14/02/2018 à 23:15, Nuria Ruiz a écrit :
>
> Hello from Analytics team:
>
> Just a brief note to announce that Wikistats 2.0 includes data about
> pageviews per project per country for the current month.
>
> Take a look, pageviews for Spanish Wikipedia this current 
> month:https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/pageviews-by-country
>
> Data is also available programatically vi APIs:
> https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Pageviews_split_by_country
>
>
> We will be deploying small UI tweaks during this week but please explore
> and let us know what you think.
>
> Thanks,
>
> Nuria
> ___
> Wikitech-l mailing 
> listWikitech-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

Trying again:

>Sorry, I can't. I opened the link you gave on hewiki. Changed from map view
>to table view. I get a list of countries - US, France, Spain, --, Japan,
>and so on. It's a link, and clicking it opens unexisting wiki article with
>the same name.

Issue should be fixed with this ticket:
https://phabricator.wikimedia.org/T187205

On Thu, Feb 15, 2018 at 9:04 AM, Nuria Ruiz <nu...@wikimedia.org> wrote:

> >Sorry, I can't. I opened the link you gave on hewiki. Changed from map
> view
> >to table view. I get a list of countries - US, France, Spain, --, Japan,
> >and so on. It's a link, and clicking it opens unexisting wiki article with
> >the same name.
>
> I see, issue should be fixed here: Sorry, I can't. I opened the link you
> gave on hewiki. Changed from map view
> to table view. I get a list of countries - US, France, Spain, --, Japan,
> and so on. It's a link, and clicking it opens unexisting wiki article with
> the same name.
>
> On Wed, Feb 14, 2018 at 4:35 PM, יגאל חיטרון <khit...@gmail.com> wrote:
>
>> Sorry, I can't. I opened the link you gave on hewiki. Changed from map
>> view
>> to table view. I get a list of countries - US, France, Spain, --, Japan,
>> and so on. It's a link, and clicking it opens unexisting wiki article with
>> the same name.
>> Igal
>>
>>
>> On Feb 15, 2018 02:23, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>>
>> > Hello,
>> >
>> > >Hello and thank you.
>> > >What do you mean in country named "--"?
>> >
>> > Not sure what you are asking, maybe a screenshot would help?
>> >
>> > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com> wrote:
>> >
>> > > Hello and thank you.
>> > > What do you mean in country named "--"?
>> > > Igal (User:IKhitron)
>> > >
>> > >
>> > > On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>> > >
>> > > Hello from Analytics team:
>> > >
>> > > Just a brief note to announce that Wikistats 2.0 includes data about
>> > > pageviews per project per country for the current month.
>> > >
>> > > Take a look, pageviews for Spanish Wikipedia this current month:
>> > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
>> > > pageviews-by-country
>> > >
>> > > Data is also available programatically vi APIs:
>> > >
>> > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
>> > > Pageviews#Pageviews_split_by_country
>> > >
>> > > We will be deploying small UI tweaks during this week but please
>> explore
>> > > and let us know what you think.
>> > >
>> > > Thanks,
>> > >
>> > > Nuria
>> > > ___
>> > > Wikitech-l mailing list
>> > > Wikitech-l@lists.wikimedia.org
>> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> > > ___
>> > > Wikitech-l mailing list
>> > > Wikitech-l@lists.wikimedia.org
>> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> > ___
>> > Wikitech-l mailing list
>> > Wikitech-l@lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>Sorry, I can't. I opened the link you gave on hewiki. Changed from map view
>to table view. I get a list of countries - US, France, Spain, --, Japan,
>and so on. It's a link, and clicking it opens unexisting wiki article with
>the same name.

I see, issue should be fixed here: Sorry, I can't. I opened the link you
gave on hewiki. Changed from map view
to table view. I get a list of countries - US, France, Spain, --, Japan,
and so on. It's a link, and clicking it opens unexisting wiki article with
the same name.

On Wed, Feb 14, 2018 at 4:35 PM, יגאל חיטרון <khit...@gmail.com> wrote:

> Sorry, I can't. I opened the link you gave on hewiki. Changed from map view
> to table view. I get a list of countries - US, France, Spain, --, Japan,
> and so on. It's a link, and clicking it opens unexisting wiki article with
> the same name.
> Igal
>
>
> On Feb 15, 2018 02:23, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>
> > Hello,
> >
> > >Hello and thank you.
> > >What do you mean in country named "--"?
> >
> > Not sure what you are asking, maybe a screenshot would help?
> >
> > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com> wrote:
> >
> > > Hello and thank you.
> > > What do you mean in country named "--"?
> > > Igal (User:IKhitron)
> > >
> > >
> > > On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
> > >
> > > Hello from Analytics team:
> > >
> > > Just a brief note to announce that Wikistats 2.0 includes data about
> > > pageviews per project per country for the current month.
> > >
> > > Take a look, pageviews for Spanish Wikipedia this current month:
> > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> > > pageviews-by-country
> > >
> > > Data is also available programatically vi APIs:
> > >
> > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> > > Pageviews#Pageviews_split_by_country
> > >
> > > We will be deploying small UI tweaks during this week but please
> explore
> > > and let us know what you think.
> > >
> > > Thanks,
> > >
> > > Nuria
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>It's just great! It took me a while to understand the meaning of
>100M->1G and friends.
Numeric localization will help a bit with that, on the way:
https://phabricator.wikimedia.org/T187010

On Thu, Feb 15, 2018 at 3:00 AM, Arturo Borrero <aborr...@wikimedia.org>
wrote:

> On Wed, Feb 14, 2018 at 11:15 PM, Nuria Ruiz <nu...@wikimedia.org> wrote:
> > Hello from Analytics team:
> >
> > Just a brief note to announce that Wikistats 2.0 includes data about
> > pageviews per project per country for the current month.
> >
> > Take a look, pageviews for Spanish Wikipedia this current month:
> > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> pageviews-by-country
> >
> > Data is also available programatically vi APIs:
> >
> > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> Pageviews#Pageviews_split_by_country
> >
> > We will be deploying small UI tweaks during this week but please explore
> > and let us know what you think.
> >
>
> It's just great! It took me a while to understand the meaning of
> 100M->1G and friends.
>
> warning: exploring too much destroys productivity :-)
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

>Nuria, well done, looks awesome!  I think you want to set zoom limits -
>right now you can zoom in and zoom out almost infinitely.
Fix on the way:
https://phabricator.wikimedia.org/T187205

On Wed, Feb 14, 2018 at 6:24 PM, Yuri Astrakhan <yuriastrak...@gmail.com>
wrote:

> Nuria, well done, looks awesome!  I think you want to set zoom limits -
> right now you can zoom in and zoom out almost infinitely. Congrats!
>
> On Wed, Feb 14, 2018 at 8:47 PM, Michael Schönitzer <
> michael.schoenit...@wikimedia.de> wrote:
>
> > here a screenshot:
> >
> > There's a second anomaly: "SX" is also listed…
> >
> > 2018-02-15 1:35 GMT+01:00 יגאל חיטרון <khit...@gmail.com>:
> >
> > > Sorry, I can't. I opened the link you gave on hewiki. Changed from map
> > view
> > > to table view. I get a list of countries - US, France, Spain, --,
> Japan,
> > > and so on. It's a link, and clicking it opens unexisting wiki article
> > with
> > > the same name.
> > > Igal
> > >
> > >
> > > On Feb 15, 2018 02:23, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > >Hello and thank you.
> > > > >What do you mean in country named "--"?
> > > >
> > > > Not sure what you are asking, maybe a screenshot would help?
> > > >
> > > > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com>
> > wrote:
> > > >
> > > > > Hello and thank you.
> > > > > What do you mean in country named "--"?
> > > > > Igal (User:IKhitron)
> > > > >
> > > > >
> > > > > On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
> > > > >
> > > > > Hello from Analytics team:
> > > > >
> > > > > Just a brief note to announce that Wikistats 2.0 includes data
> about
> > > > > pageviews per project per country for the current month.
> > > > >
> > > > > Take a look, pageviews for Spanish Wikipedia this current month:
> > > > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> > > > > pageviews-by-country
> > > > >
> > > > > Data is also available programatically vi APIs:
> > > > >
> > > > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> > > > > Pageviews#Pageviews_split_by_country
> > > > >
> > > > > We will be deploying small UI tweaks during this week but please
> > > explore
> > > > > and let us know what you think.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Nuria
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> >
> >
> >
> > --
> > Michael F. Schönitzer
> >
> >
> >
> > Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> > Tel. (030) 219 158 26-0
> > http://wikimedia.de
> >
> > Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> > Wissens frei teilhaben kann. Helfen Sie uns dabei!
> > http://spenden.wikimedia.de/
> >
> > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
> > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter
> > der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> > Körperschaften I Berlin, Steuernummer 27/681/51985.
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

Hello,

>Hello and thank you.
>What do you mean in country named "--"?

Not sure what you are asking, maybe a screenshot would help?

On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון <khit...@gmail.com> wrote:

> Hello and thank you.
> What do you mean in country named "--"?
> Igal (User:IKhitron)
>
>
> On Feb 15, 2018 00:15, "Nuria Ruiz" <nu...@wikimedia.org> wrote:
>
> Hello from Analytics team:
>
> Just a brief note to announce that Wikistats 2.0 includes data about
> pageviews per project per country for the current month.
>
> Take a look, pageviews for Spanish Wikipedia this current month:
> https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> pageviews-by-country
>
> Data is also available programatically vi APIs:
>
> https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> Pageviews#Pageviews_split_by_country
>
> We will be deploying small UI tweaks during this week but please explore
> and let us know what you think.
>
> Thanks,
>
> Nuria
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Wikistats 2.0 - Now with Maps!

[Nuria Ruiz]

Hello from Analytics team:

Just a brief note to announce that Wikistats 2.0 includes data about
pageviews per project per country for the current month.

Take a look, pageviews for Spanish Wikipedia this current month:
https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/pageviews-by-country

Data is also available programatically vi APIs:

https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Pageviews_split_by_country

We will be deploying small UI tweaks during this week but please explore
and let us know what you think.

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats gets a facelift - Alpha Launch of Wikistats 2

[Nuria Ruiz]

>Is showing "all project for a specific language" somewhere in the
remaining roadmap?
Let me make sure I understand: Showing metrics in a language-centric
fashion rather than project-centric fashion?

No, it is not something we are considering for the near future (not that is
not doable, it is just not a priority). Our future work will be around
these general areas: "pageviews per project per country" (soon), "mobile
layouts" (in some months), "localization" (mid term)  and invisible work in
the backend that help us process this much data in less time (all the time,
basically).

On Fri, Dec 15, 2017 at 12:30 PM, mathieu stumpf guntz <
psychosl...@culture-libre.org> wrote:

> Yes this is really already in a great shape, congratulations and thank you.
>
> Is showing "all project for a specific language" somewhere in the
> remaining roadmap?
>
>
>
> Le 14/12/2017 à 21:41, zppix e a écrit :
>
>> Great Work I love the design. Can't wait for finished product!
>>
>> --
>> Zppix
>> Volunteer Wikimedia Developer
>> Volunteer Wikimedia GCI2017 Mentor
>> enwp.org/User:Zppix
>> **Note: I do not work for Wikimedia Foundation, or any of its chapters.**
>>
>> On Dec 14, 2017, at 1:17 PM, Jonathan Morgan <jmor...@wikimedia.org>
>>> wrote:
>>>
>>> This is fabulous! Thank you, Erik Zachte, Analytics team, and everyone
>>> else
>>> involved in this project for giving us the powerful, usable stats
>>> dashboard
>>> we deserve :)
>>>
>>> - J
>>>
>>> On Thu, Dec 14, 2017 at 5:10 AM, Niharika Kohli <nko...@wikimedia.org>
>>> wrote:
>>>
>>> This is awesome. Great job A-team!
>>>>
>>>> On Thu, Dec 14, 2017 at 12:12 PM, Victoria Coleman <
>>>> vcole...@wikimedia.org
>>>> wrote:
>>>>
>>>> Nuria and team, fabulous work!  Wikistats 2 is such a huge improvement!
>>>>> Thank you!
>>>>>
>>>>>
>>>>> Best wishes,
>>>>>
>>>>> Victoria Coleman
>>>>>
>>>>> Chief Technology Officer
>>>>> Wikimedia Foundation
>>>>> 1 Montgomery Street, Suite 1600
>>>>> San Francisco, CA 94104
>>>>>
>>>>> +1-650-703-8112
>>>>>
>>>>> vcole...@wikimedia.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Dec 13, 2017, at 8:26 PM, Nuria Ruiz <nu...@wikimedia.org> wrote:
>>>>>>
>>>>>> Hello from Analytics Team!
>>>>>>
>>>>>> We are happy to announce the Alpha release of Wikistats 2. Wikistats
>>>>>>
>>>>> has
>>>>
>>>>> been redesigned for architectural simplicity, faster data processing,
>>>>>>
>>>>> and a
>>>>>
>>>>>> more dynamic and interactive user experience. First goal is to match
>>>>>>
>>>>> the
>>>>
>>>>> numbers of the current system, and to provide the most important
>>>>>>
>>>>> reports,
>>>>
>>>>> as decided by the Wikistats community (see survey) [1].  Over time, we
>>>>>>
>>>>> will
>>>>>
>>>>>> continue to migrate reports and add new ones that you find useful. We
>>>>>>
>>>>> can
>>>>
>>>>> also analyze the data in new and interesting ways, and look forward to
>>>>>> hearing your feedback and suggestions. [2]
>>>>>>
>>>>>> You can go directly to Spanish Wikipedia
>>>>>> https://stats.wikimedia.org/v2/#/es.wikipedia.org
>>>>>>
>>>>>> or browse all projects
>>>>>> https://stats.wikimedia.org/v2/#/all-projects
>>>>>>
>>>>>> The new site comes with a whole new set of APIs, similar to our
>>>>>>
>>>>> existing
>>>>
>>>>> Pageview API but with edit data. You can start using them today, they
>>>>>>
>>>>> are
>>>>
>>>>> documented here:
>>>>>>
>>>>>> https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats
>>>>>>
>>>>>>
>>>

[Wikitech-l] Wikistats gets a facelift - Alpha Launch of Wikistats 2

[Nuria Ruiz]

Hello from Analytics Team!

We are happy to announce the Alpha release of Wikistats 2. Wikistats has
been redesigned for architectural simplicity, faster data processing, and a
more dynamic and interactive user experience. First goal is to match the
numbers of the current system, and to provide the most important reports,
as decided by the Wikistats community (see survey) [1].  Over time, we will
continue to migrate reports and add new ones that you find useful. We can
also analyze the data in new and interesting ways, and look forward to
hearing your feedback and suggestions. [2]

You can go directly to Spanish Wikipedia
https://stats.wikimedia.org/v2/#/es.wikipedia.org

or browse all projects
https://stats.wikimedia.org/v2/#/all-projects

The new site comes with a whole new set of APIs, similar to our existing
Pageview API but with edit data. You can start using them today, they are
documented here:

https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats


FAQ:

Why is this an alpha?
There are features that we feel a full-fledged product should have that are
still missing, such as localization. The data-processing pipeline for the
new Wikistats has been rebuilt from scratch (it uses distributed-computing
tools such as Hadoop) and we want to see how it is used before calling it
final. Also while we aim to update data monthly, it will happen a few days
after the month rolls because of the amount of data to move and compute.

How about comparing data between two wikis?
You can do it with two tabs but we are aware this UI might not solve all
use cases for the most advanced Wikistats users. We aim to tackle those in
the future.

How do I file bugs?
Use the handy link in the footer:
https://phabricator.wikimedia.org/maniphest/task/edit/?title=Wikistats%20Bug=Analytics-Wikistats,Analytics

How do I comment on design?
The consultation on design already happened but we are still watching the
talk page:
https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback/Round2


[1]
https://www.mediawiki.org/wiki/Analytics/Wikistats/DumpReports/Future_per_report
[2] https://wikitech.wikimedia.org/wiki/Talk:Analytics/Systems/Wikistats
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] Drop in mainpage pageviews?

[Nuria Ruiz]

>It was explained as one that changes it a lot. Maybe there was a wrong
explanation.

Sorry, not sure what you mean, can you link to any documentation we can
maybe clarify? As the measure two metrics "Unique Devices" and "Pageviews"
is completely different.

On Mon, Jul 17, 2017 at 11:05 AM, יגאל חיטרון <khit...@post.bgu.ac.il>
wrote:

> It was explained as one that changes it a lot. Maybe there was a wrong
> explanation.
> Igal
>
> 2017-07-17 20:52 GMT+03:00 Nuria Ruiz <nu...@wikimedia.org>:
>
> > >I'm talking about research "Unique devices".
> > That had no effect on pageviews. I think you are probably mixing two
> > different projects.
> >
> >
> > As I mentioned pageview counting has not changed for more than 2 years.
> >
> >
> > Thanks,
> >
> >
> > Nuria
> >
> > On Mon, Jul 17, 2017 at 9:43 AM, יגאל חיטרון <khit...@post.bgu.ac.il>
> > wrote:
> >
> > > Hi. I do not remember when did this happen - this april or two years
> ago.
> > > I'm talking about research "Unique devices".
> > > Igal
> > >
> > >
> > > 2017-07-17 19:37 GMT+03:00 Nuria Ruiz <nu...@wikimedia.org>:
> > >
> > > > > do not remember the exact date, but a couple of months ago the way
> > for
> > > > > pageviews counting was changed, using cookies, affecting the mobile
> > web
> > > > > views. This can be the cause.
> > > > > Igal (User IKhitron)
> > > >
> > > > ... not sure what you are referring to, can you be more specific?
> > We
> > > > have not changed the way we measure pageviews for 2 years
> > > >
> > > > There are improvements/fixes continously done to the system and those
> > get
> > > > documented here:
> > > >
> > > > https://meta.wikimedia.org/wiki/Research:Page_view#Change_log
> > > >
> > > >
> > > > FYI, we have opened a ticket for this issue:
> > > > https://phabricator.wikimedia.org/T170845
> > > >
> > > > Without looking at it in detail there are many possible causes:
> removal
> > > of
> > > > "bad" pageviews (like requests for banners) or drop of traffic due to
> > > > having less bots.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Jul 15, 2017 at 11:38 PM, Strainu <strain...@gmail.com>
> wrote:
> > > >
> > > > > 2017-07-15 14:34 GMT+03:00 יגאל חיטרון <khit...@gmail.com>:
> > > > > > Hello, Strainu.
> > > > > > 1. Try in place of "all-access" use different platforms. You'll
> > see,
> > > > as I
> > > > > > expected reading your letter, that the effect you recognized
> > appears
> > > in
> > > > > > "mobile-web" mostly.
> > > > > > 2. I do not remember the exact date, but a couple of months ago
> the
> > > way
> > > > > for
> > > > > > pageviews counting was changed, using cookies, affecting the
> mobile
> > > web
> > > > > > views. This can be the cause.
> > > > > > Igal (User IKhitron)
> > > > >
> > > > > Thank you Igal, that must be the reason, I'll look for the change
> > > > > announcement just to understand what it means exactly.
> > > > >
> > > > > Strainu
> > > > >
> > > > > >
> > > > > > On Jul 15, 2017 13:47, "Strainu" <strain...@gmail.com> wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Starting from an unrelated discussion on meta, I noticed a
> > > significant
> > > > > >> drop in main page views for several wikis starting from April
> this
> > > > > >> year. Is there anything we (or Google) did at that time to
> justify
> > > > > >> this drop?
> > > > > >>
> > > > > >> ro.wiki: http://tools.wmflabs.org/pageviews/?project=ro.
> > > > > >> wikipedia.org=all-access=user=2016-
> > > > > >> 01=2017-06=Pagina_principal%C4%83
> > > 

[Wikitech-l] More Detailed Browser Stats for Desktop Sites

[Nuria Ruiz]

Hello:


Please take a look at the new browser report with more detailed desktop
site data (all wikimedia projects agreggated):

https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-browser

Some highlights:

* Data is very stable over the last year

* Chrome in the lead with 45% of traffic, closely followed by IE (18%) and
FF (13%)

* The bulk of IE traffic is IE11 and IE7

* Edge shows up with 4% slowly catching up to Safari (5%)

* This data is still subject to fluctuations due to bot traffic not
identified as such.  We will be working on this next year.


Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] Drop in mainpage pageviews?

[Nuria Ruiz]

>I'm talking about research "Unique devices".
That had no effect on pageviews. I think you are probably mixing two
different projects.


As I mentioned pageview counting has not changed for more than 2 years.


Thanks,


Nuria

On Mon, Jul 17, 2017 at 9:43 AM, יגאל חיטרון <khit...@post.bgu.ac.il> wrote:

> Hi. I do not remember when did this happen - this april or two years ago.
> I'm talking about research "Unique devices".
> Igal
>
>
> 2017-07-17 19:37 GMT+03:00 Nuria Ruiz <nu...@wikimedia.org>:
>
> > > do not remember the exact date, but a couple of months ago the way for
> > > pageviews counting was changed, using cookies, affecting the mobile web
> > > views. This can be the cause.
> > > Igal (User IKhitron)
> >
> > ... not sure what you are referring to, can you be more specific? We
> > have not changed the way we measure pageviews for 2 years
> >
> > There are improvements/fixes continously done to the system and those get
> > documented here:
> >
> > https://meta.wikimedia.org/wiki/Research:Page_view#Change_log
> >
> >
> > FYI, we have opened a ticket for this issue:
> > https://phabricator.wikimedia.org/T170845
> >
> > Without looking at it in detail there are many possible causes: removal
> of
> > "bad" pageviews (like requests for banners) or drop of traffic due to
> > having less bots.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Jul 15, 2017 at 11:38 PM, Strainu <strain...@gmail.com> wrote:
> >
> > > 2017-07-15 14:34 GMT+03:00 יגאל חיטרון <khit...@gmail.com>:
> > > > Hello, Strainu.
> > > > 1. Try in place of "all-access" use different platforms. You'll see,
> > as I
> > > > expected reading your letter, that the effect you recognized appears
> in
> > > > "mobile-web" mostly.
> > > > 2. I do not remember the exact date, but a couple of months ago the
> way
> > > for
> > > > pageviews counting was changed, using cookies, affecting the mobile
> web
> > > > views. This can be the cause.
> > > > Igal (User IKhitron)
> > >
> > > Thank you Igal, that must be the reason, I'll look for the change
> > > announcement just to understand what it means exactly.
> > >
> > > Strainu
> > >
> > > >
> > > > On Jul 15, 2017 13:47, "Strainu" <strain...@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Starting from an unrelated discussion on meta, I noticed a
> significant
> > > >> drop in main page views for several wikis starting from April this
> > > >> year. Is there anything we (or Google) did at that time to justify
> > > >> this drop?
> > > >>
> > > >> ro.wiki: http://tools.wmflabs.org/pageviews/?project=ro.
> > > >> wikipedia.org=all-access=user=2016-
> > > >> 01=2017-06=Pagina_principal%C4%83
> > > >>
> > > >> hu.wiki: http://tools.wmflabs.org/pageviews/?project=hu.
> > > >> wikipedia.org=all-access=user=2016-
> > > >> 01=2017-06=Kezd%C5%91lap
> > > >>
> > > >> fr.wiki: http://tools.wmflabs.org/pageviews/?project=fr.
> > > >> wikipedia.org=all-access=user=2016-
> > > >> 01=2017-06=Wikip%C3%A9dia:Accueil_principal
> > > >>
> > > >> de.wiki: http://tools.wmflabs.org/pageviews/?project=de.
> > > >> wikipedia.org=all-access=user=2016-
> > > >> 01=2017-06=Wikipedia:Hauptseite
> > > >>
> > > >> en.wiki: http://tools.wmflabs.org/pageviews/?project=en.
> > > >> wikipedia.org=all-access=user=2016-
> > > >> 01=2017-06=Main_Page
> > > >> (slightly different pattern)
> > > >>
> > > >> The same cannot be said for other projects, for instance uk.wiki:
> > > >> http://tools.wmflabs.org/pageviews/?project=uk.
> > > wikipedia.org=all-
> > > >> access=user=2016-01=2017-06=%D0%93%
> > > >> D0%BE%D0%BB%D0%BE%D0%B2%D0%BD%D0%B0_%D1%81%D1%82%D0%BE%D1%
> > > >> 80%D1%96%D0%BD%D0%BA%D0%B0
> > > >>
> > > >> Thanks,
> > > >>Strainu
> > > >>
> > > >> ___
> > > >> Wikitech-l mailing list
> > > >> Wikitech-l@lists.wikimedia.org
> > > >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > > ___
> > > Analytics mailing list
> > > analyt...@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/analytics
> > >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] Drop in mainpage pageviews?

[Nuria Ruiz]

> do not remember the exact date, but a couple of months ago the way for
> pageviews counting was changed, using cookies, affecting the mobile web
> views. This can be the cause.
> Igal (User IKhitron)

... not sure what you are referring to, can you be more specific? We
have not changed the way we measure pageviews for 2 years

There are improvements/fixes continously done to the system and those get
documented here:

https://meta.wikimedia.org/wiki/Research:Page_view#Change_log


FYI, we have opened a ticket for this issue:
https://phabricator.wikimedia.org/T170845

Without looking at it in detail there are many possible causes: removal of
"bad" pageviews (like requests for banners) or drop of traffic due to
having less bots.











On Sat, Jul 15, 2017 at 11:38 PM, Strainu  wrote:

> 2017-07-15 14:34 GMT+03:00 יגאל חיטרון :
> > Hello, Strainu.
> > 1. Try in place of "all-access" use different platforms. You'll see, as I
> > expected reading your letter, that the effect you recognized appears in
> > "mobile-web" mostly.
> > 2. I do not remember the exact date, but a couple of months ago the way
> for
> > pageviews counting was changed, using cookies, affecting the mobile web
> > views. This can be the cause.
> > Igal (User IKhitron)
>
> Thank you Igal, that must be the reason, I'll look for the change
> announcement just to understand what it means exactly.
>
> Strainu
>
> >
> > On Jul 15, 2017 13:47, "Strainu"  wrote:
> >
> >> Hi,
> >>
> >> Starting from an unrelated discussion on meta, I noticed a significant
> >> drop in main page views for several wikis starting from April this
> >> year. Is there anything we (or Google) did at that time to justify
> >> this drop?
> >>
> >> ro.wiki: http://tools.wmflabs.org/pageviews/?project=ro.
> >> wikipedia.org=all-access=user=2016-
> >> 01=2017-06=Pagina_principal%C4%83
> >>
> >> hu.wiki: http://tools.wmflabs.org/pageviews/?project=hu.
> >> wikipedia.org=all-access=user=2016-
> >> 01=2017-06=Kezd%C5%91lap
> >>
> >> fr.wiki: http://tools.wmflabs.org/pageviews/?project=fr.
> >> wikipedia.org=all-access=user=2016-
> >> 01=2017-06=Wikip%C3%A9dia:Accueil_principal
> >>
> >> de.wiki: http://tools.wmflabs.org/pageviews/?project=de.
> >> wikipedia.org=all-access=user=2016-
> >> 01=2017-06=Wikipedia:Hauptseite
> >>
> >> en.wiki: http://tools.wmflabs.org/pageviews/?project=en.
> >> wikipedia.org=all-access=user=2016-
> >> 01=2017-06=Main_Page
> >> (slightly different pattern)
> >>
> >> The same cannot be said for other projects, for instance uk.wiki:
> >> http://tools.wmflabs.org/pageviews/?project=uk.
> wikipedia.org=all-
> >> access=user=2016-01=2017-06=%D0%93%
> >> D0%BE%D0%BB%D0%BE%D0%B2%D0%BD%D0%B0_%D1%81%D1%82%D0%BE%D1%
> >> 80%D1%96%D0%BD%D0%BA%D0%B0
> >>
> >> Thanks,
> >>Strainu
> >>
> >> ___
> >> Wikitech-l mailing list
> >> Wikitech-l@lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Analytics mailing list
> analyt...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/analytics
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Legacy Pagecounts available in API

[Nuria Ruiz]

Hello!

The analytics team would like to announce that legacy pagecounts are now
available programatically in an API.

"Pagecount" is the legacy definition of what we now call "pageview".
Pagecounts agreggated per project are available on API endpoint since
January 2008 to December 2016. The main difference among pagecounts and the
current pageview data is lack of filtering of self-reported bots, thus
automated and human traffic are reported together.  You can access data
overall but also mobile/desktop split.

Note that -at this time- we still do not have pagecount data per article,
thus far we have loaded only per-project legacy data.

More info  and examples of how to query the API can be found here:
https://wikitech.wikimedia.org/wiki/Analytics/AQS/Legacy_Pagecounts

Thanks to Thomas Steiner for promptly updating the pageviews.js node client
to be able to access pagecounts. Javascript client can be found here:
https://github.com/tomayac/pageviews.js


A glimpse of this data:
https://analytics.wikimedia.org/dashboards/reportcard/#pagecounts-dec-2007-dec-2016

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Migrated Reportcard with Updated Data

[Nuria Ruiz]

Hello!

The Analytics team would like to announce that we have migrated the
reportcard to a new domain:

https://analytics.wikimedia.org/dashboards/reportcard/#
pageviews-july-2015-now

The migrated reportcard includes both legacy and current pageview data,
daily unique devices and new editors data. Pageview and devices data is
updated daily but editor data is still updated ad-hoc.

The team is working at this time on revamping the way we compute edit data
and we hope to be able to provide monthly updates for the main edit metrics
this quarter. Some of those will be visible in the reportcard but the new
wikistats will have more detailed reports.

You can follow the new wikistats project here: https://phabricator.
wikimedia.org/T130256

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Monthly page view stats that can now be queried via Pageview API.

[Nuria Ruiz]

Hello!

The Analytics team would like to announce that the Pageview API is able to
return monthly pageview stats as of this week.


For example, the request below will get you a monthly count of pageviews
for de.wikipedia's article Barack_Obama for the year 2016 (note 'monthly'
parameter on url)

http://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/de.wikipedia/all-access/all-agents/Barack_Obama/
*monthly*/2016010100/2016123100


More documentation and examples can be found on our quickstart guide:
https://wikitech.wikimedia.org/wiki/Analytics/PageviewAPI#Quick_Start

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[Nuria Ruiz]

>The organizers of the Summit propose to have a few main topics defined
>beforehand, so we can invite the people beyond the usual Summit
>participants that should be involved in these discussions.

>We don't have any opinion about which topics should be selected. However,
>we believe that it is important to choose complex topics with a high user
>impact (direct or indirect) and ramifications in multiple technical areas.
>The Summit with its +200 participants provides a great framework to push
>these topics.

I think this idea is inline with how most tech conferences work. In order
for the summit to be successful we
should know what type of event we are creating (purely technical, a venue
to explore user requests, a venue to influence
product roadmap, just "get in touch"... whatever)

Seems that defining what areas we want to cover in the summit is
a prerequisite  to do what Brion was asking for in the initial e-mail, to
open the summit beyond WMF.

On Wed, Sep 7, 2016 at 4:00 PM, Quim Gil  wrote:

> On Wed, Sep 7, 2016 at 8:45 PM, Rob Lanphier  wrote:
>
> > It's hard for me to read this as anything but "WMF Product selects the
> > topics, and then 'listens' for objections on wikitech-l".
>
>
> Since this is not even remotely what I am trying to say, let me try again,
> while I keep iterating on the proposal:
>
> The organizers of the Summit propose to have a few main topics defined
> beforehand, so we can invite the people beyond the usual Summit
> participants that should be involved in these discussions.
>
> We don't have any opinion about which topics should be selected. However,
> we believe that it is important to choose complex topics with a high user
> impact (direct or indirect) and ramifications in multiple technical areas.
> The Summit with its +200 participants provides a great framework to push
> these topics.
>
> We don't have any opinion about who decides these topics and how. However,
> we need good enough answers and fast, so we can start organizing the
> participation and the program. There will be time to fine tune things as we
> go, and there will be enough space for all the topics not selected
> beforehand, in the unconference part of the Summit.
>
> We believe that by having the right people focusing on a few main topics
> before and during the Summit, the outcome of the event will have a clearer
> impact in the strategy and goals that ultimately drive where we put a lot
> of our attention and resources.
>
> Is your sense that the
> > correct direction for us is for someone to provide more top-down
> > direction instead of wikitech-l conversation?
> >
>
> My sense is that we need to open registration, call for participation, and
> call for travel sponsorship requests as soon as possible. This list, the
> Architecture committee, the WMF Technology management team, and the WMF
> Product management team are aware of this situation and our request to
> define main topics. I will be checking with all these sources with the goal
> of having a good enough answer that allows us to move forward with the
> Summit organization.
>
> --
> Quim Gil
> Engineering Community Manager @ Wikimedia Foundation
> http://www.mediawiki.org/wiki/User:Qgil
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Consolidating dashboards and data downloads on analytics.wikimedia.org

[Nuria Ruiz]

Hello!

We would like to announce http://analytics.wikimedia.org the domain where
the analytics team hopes to consolidate dashboards and data downloads (a
work in progress)


We have two dashboards to announce with data we hope you find interesting:

1. The new traffic reports with browser data for mobile and desktop domains:
*http://tinyurl.com/zqqr3zd *

2. The vital signs dashboard that displays several metrics that are of
interest like Pageviews and Unique Devices:
*http://tinyurl.com/jr4oou3 *

Thanks,

Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] 'Unique Devices' Data Visualizations Available

[Nuria Ruiz]

Hello!


The analytics team would like to announce that we have a new visualization
for Unique Devices data. As you know Unique Devices [1] is our best proxy
to calculate Unique Users. We would like to reiterate that the data is
available in a public API that anyone can access [2]. We calculate Uniques
daily and monthly.


See, for example, "Daily Unique Devices" for Spanish Wikipedia versus
French wikipedia:
https://vital-signs.wmflabs.org/#projects=frwiki,eswiki/metrics=UniqueDevices

FYI that dashboard would not work on IE, only on Edge.

Thanks,

Nuria

[1] https://meta.wikimedia.org/wiki/Research:Unique_Devices
[2] https://wikitech.wikimedia.org/wiki/Analytics/Unique_Devices
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] Unique Devices data available on API

[Nuria Ruiz]

>Sure, and browses which rejects or periodically delete cookies will be
counted multiple times
Actually no, they will be counted only once towards the period in which we
are counting them (daily or monthly) as long as the IP of the device is the
same. Not multiple times. In mobile IPs are shared due to NAT-ing and thus,
this method undercounts.
This is explained in detail here:
https://wikitech.wikimedia.org/wiki/Analytics/Unique_Devices/Last_access_solution#Nocookie_Offset



>a device might use multiple browsers (especially the embedded browsers on
mobile devices)
This is correct, multiple browsers on one device will be counted multiple
times. This is less of a common browsing pattern than using the same method
of access every time.

On Tue, Apr 19, 2016 at 3:12 PM, Gergo Tisza  wrote:

> On Tue, Apr 19, 2016 at 11:52 PM, Erik Zachte 
> wrote:
>
>> Like Nuria said: this is unique devices, not unique people. Many people
>> in the Global North use more than one device to access Wikipedia (desktop,
>> tablet, phone).
>>
>
> Sure, and browses which rejects or periodically delete cookies will be
> counted multiple times, and a device might use multiple browsers
> (especially the embedded browsers on mobile devices)... even taking all
> that into account, a factor of four seems like a large disparity between
> this data and the audience panel numbers.
>
> ___
> Analytics mailing list
> analyt...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/analytics
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Unique Devices data available on API

[Nuria Ruiz]

Hello!

The analytics team is happy to announce that the Unique Devices data is now
available to be queried programmatically via an API.

This means that getting the daily number of unique devices [1] for English
Wikipedia for the month of February 2016, for all sites (desktop and
mobile) is as easy as launching this query:

https://wikimedia.org/api/rest_v1/metrics/unique-devices/en.wikipedia.org/all-sites/daily/20160201/20160229

You can get started by taking a look at our docs:
https://wikitech.wikimedia.org/wiki/Analytics/Unique_Devices#Quick_Start

If you are not familiar with the Unique Devices data the main thing you
need to know is that
is a good proxy metric to measure Unique Users, more info below.

Since 2009, the Wikimedia Foundation used comScore to report data about
unique web visitors.  In January 2016, however, we decided to stop
reporting comScore numbers [2] because of certain limitations in the
methodology, these limitations translated into misreported mobile usage. We
are now ready to replace comscore numbers with the Unique Devices Dataset .
While unique devices does not equal unique visitors, it is a good proxy for
that metric, meaning that a major increase in the number of unique devices
is likely to come from an increase in distinct users. We understand that
counting uniques raises fairly big privacy concerns and we use a very
private conscious way to count unique devices, it does not include any
cookie by which your browser history can be tracked [3].


[1] https://meta.wikimedia.org/wiki/Research:Unique_Devices
[2] [https://meta.wikimedia.org/wiki/ComScore/Announcement
[3]
https://meta.wikimedia.org/wiki/Research:Unique_Devices#How_do_we_count_unique_
devices.3F
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Transitioning wikistats pageview reports to use new pageview definition

[Nuria Ruiz]

Hello!

The analytics team wishes to announce that we have finally transitioned
several of the pageview reports in stats.wikimedia.org  to the new pageview
definition [1]. This means that we should no longer have two conflicting
sources of pageview numbers.

While we are not not fully done transitioning pageview reports we feel this
is an important enough milestone that warrants some communication. BIG
Thanks to Erik Z. for his work on this project.

Please take a look at a report using the new definition (a banner is
present when report has been updated)
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm

Thanks,

Nuria


[1] https://meta.wikimedia.org/wiki/Research:Page_view
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Fwd: Phabricator monthly statistics - 2015-01

[Nuria Ruiz]

Also keep in mind that it is used by many people to track feature-requests
(long and short-term, planned and wishlist), as well as bugs, so some
growth is both inevitable and desired. (I believe this is what James is
saying)
Much agreed. Phabricator is used to keep track of many other things besides
bugs, in our team we keep our whole backlog there thus it is expected to
grow at a much faster rate than a bug list would. A backlog is a repository
of ideas and some of those are clearly defined tasks but not others.



On Mon, Feb 2, 2015 at 1:02 AM, Nick Wilson (Quiddity) 
nwil...@wikimedia.org wrote:

 On Sun, Feb 1, 2015 at 2:16 PM, Pine W wiki.p...@gmail.com wrote:

  Hi Quim,
 
  Thanks. Just wondering, do we have a graph somewhere that shows trends?
  There is quite a gap between tasks created and tasks closed, and if
 that's
  normal each month then there needs to be some thought about how to fix
 the
  gap, yes?
 
  Thanks,
  Pine
 


 Also keep in mind that it is used by many people to track feature-requests
 (long and short-term, planned and wishlist), as well as bugs, so some
 growth is both inevitable and desired. (I believe this is what James is
 saying). But I agree that an increase of old-bug triage and resolving,
 would be welcomed.
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Investigating building an apps content service using RESTBase and Node.js

[Nuria Ruiz]

 In general I'm in favor of more ad-hoc project-specific teams rather than
completely siloing every service to the Services group, or every mobile UI
to the Mobile group.=
Agreed, as long as everyone deploying services communicates through
services team so there is no duplication of solutions.
We should have 1 routing layer, standard monitoring and standard caching as
it is been pointed out before. Specifics of services might be different but
it seems the core mission of service team to provide common infrastructure,
right?.
That means that the 1st service we develop will be more work for services
team than subsequent services. *Ideally* the mobile team should develop the
service without having to solve (for example) any caching problems that are
not mobile specific.





On Wed, Feb 4, 2015 at 3:04 PM, James Douglas jdoug...@wikimedia.org
wrote:

  In general I'm in favor of more ad-hoc project-specific teams rather than
 completely siloing every service to the Services group, or every mobile UI
 to the Mobile group.

 I strongly agree.  Based on experience on both sides of this spectrum, I
 recommend (when feasible) favoring feature teams over functional teams.

 On Wed, Feb 4, 2015 at 3:00 PM, Brion Vibber bvib...@wikimedia.org
 wrote:

  I think the way we'd want to go is roughly to have a *partnership
 between*
  the Services and Mobile teams produce and maintain the service.
 
  (Note that the state of the art is that Mobile Apps are using Mobile
 Web's
  MobileFrontend extension as an intermediate API to aggregate  format
 page
  data -- which basically means Max has done the server-side API work for
  Mobile Apps so far.)
 
  I'd expect to see Max and/or someone else from the Mobile team
  collaborating with the Services team to create what y'all need:
  1) something that does what Mobile Apps needs it to...
  2) and can be maintained like Services needs it to.
 
  In general I'm in favor of more ad-hoc project-specific teams rather than
  completely siloing every service to the Services group, or every mobile
 UI
  to the Mobile group.
 
  -- brion
 
  On Wed, Feb 4, 2015 at 2:29 PM, Corey Floyd cfl...@wikimedia.org
 wrote:
 
   On Wed, Feb 4, 2015 at 11:41 AM, Gabriel Wicke gwi...@wikimedia.org
   wrote:
  
Regarding general-purpose APIs vs. mobile: I think mobile is in some
   ways a
special case as their content transformation needs are closely
 coupled
   with
the way the apps are presenting the content. Additionally, at least
  until
SPDY is deployed there is a strong performance incentive to bundle
information in a single response tailored to the app's needs. One
   strategy
employed by Netflix is to introduce a second API layer

   
  
 
 http://techblog.netflix.com/2012/07/embracing-differences-inside-netflix.html

on
top of the general content API to handle device-specific needs. I
 think
this is a sound strategy, as it contains the volatility in a separate
   layer
while ensuring that everything is ultimately consuming the
   general-purpose
API. If the need for app-specific massaging disappears over time, we
  can
simply shut down the custom service / API end point without affecting
  the
general API.
   
  
  
   I can definitely understand that motivation for providing mobile
 specific
   service layer - so if the services team wants to implement the API in
  this
   way and support that architecture, I am totally on board.
  
   My remaining hesitation here is that from the reading of this proposal,
  the
   mobile team is the owner of implementing this service, not the services
   team (Maybe I am misreading?).
  
   This leads me to ask questions like:
   Why is the mobile apps team investigating which is the best server side
   technology? That seems outside of our domain knowledge.
   Who will be responsible for maintaining this code?
   Who will be testing it to make sure that is performant?
  
   I'm new, so maybe these answers are obvious to others, but to me they
  seem
   fuzzy when responsibilities are divided between two teams.
  
   I would propose that this be a project that the Services Team owns. And
   that the Mobile Apps Team defines specs on what they need the new
 service
   to provide.
   ___
   Wikitech-l mailing list
   Wikitech-l@lists.wikimedia.org
   https://lists.wikimedia.org/mailman/listinfo/wikitech-l
  
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] internacionalization for client and server side integrated with handlebars

[Nuria Ruiz]

Hello,

Those of you that have deal with client side translation in javascript
might appreciate this library recently released from yahoo:

http://formatjs.io/github/


Just an FYI


Nuria
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [WikimediaMobile] need for an app? (was: Future platforms, devices, and consumers)

[Nuria Ruiz]

Offline storage is hard in a browser, as you pointed out; that's too much
detail for me to understand quickly, and I have no comment yet. In
principle, such concern is valid.
Even the w3 held a working group for a while that was around how broken app
cache was (http://www.w3.org/community/fixing-appcache/)

It is still broken and been so for many years, well documented since 2011:
http://www.w3.org/2011/web-apps-ws/papers/Facebook.html




On Sun, Aug 17, 2014 at 5:33 AM, svetlana svetl...@fastmail.com.au wrote:

 Dmitry Brant wrote:
  The fact that you don't see the benefits of the native app over the
 mobile
  website is simply an indication that we still have a lot of work to do
 with
  the apps, which we are excited to do.

 Partly this is because you don't support my mobile platform.

 Dmitry Brant wrote:
  But, is it a waste of effort to bring a truly integrated, seamless
  Wikipedia experience to our users' mobile devices?  I don't think so.
  Nor
  is it a waste of effort for the WMF to be seen as a driving force in
 mobile
  design and mobile user experience.

 I was assuming that integration and being seamless are easily doable from
 a web browser.

 Offline storage is hard in a browser, as you pointed out; that's too much
 detail for me to understand quickly, and I have no comment yet. In
 principle, such concern is valid.

 Documenting extra differences and shortcomings of web browsers could be a
 nice task.

 svetlana

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Requests with data URIs appended to proper URLs

[Nuria Ruiz]

I can see those images in the CSS  file that results after this call as
background images on the default skin of es.wikipedia. They look correct in
the CSS:

http://bits.wikimedia.org/es.wikipedia.org/load.php?debug=falselang=esmodules=ext.gadget.a-commons-directo%2Cimagenesinfobox%2CrefToolbar%7Cext.uls.nojs%7Cext.visualEditor.viewPageTarget.noscript%7Cext.wikihiero%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.interface%7Cmediawiki.ui.button%7Cskins.vector.stylesonly=stylesskin=vector*


The likely explanation could be that there is some syntax issue (that only
appears on windows?) on the data image path specified by us in the css and
thus some less forgiving browsers are not interpreting the image in the css
as a data url but rather as a regular one and then the url is composed as
if it was a relative url (using the current domain) and fetched (or tried
to fetch)

That is, some browsers on windows are interpreting data urls as if they
were like this:
background-url:url('/some/relative/path')

In any case that fetch should be a 404 so the thing we should probably
think of fixing  going forward is not counting 404 urls in pageviews.






On Thu, Jun 5, 2014 at 6:12 PM, Bartosz Dziewoński matma@gmail.com
wrote:

 On Thu, 05 Jun 2014 15:36:07 +0200, Christian Aistleitner 
 christ...@quelltextlich.at wrote:

  The image data in the data uri scheme decodes to images from
 VectorBeta [3] like:
  VectorBeta/resources/typography/images/search-fade.png
   VectorBeta/resources/typography/images/tab-break.png
   VectorBeta/resources/typography/images/tab-current-fade.png
   VectorBeta/resources/typography/images/portal-break.png


 These images are also part of the core Vector skin, where
 they sit at [mediawiki/core]/skins/vector/images.

 --
 Matma Rex

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Is there a way to visualize where come from editors of a specific page?

[Nuria Ruiz]

Adding analytics list to make sure everyone in the team sees this thread.

Is there a way to visualize where editors of a specific page come from?
I assume you mean if this data is available to the general public. By
reading the couple links you posted seems like the consensus was that this
data is too private to be made public and thus, it is only accessible
inside WMF or to users with CheckUser rights.

As far as we know nothing has changed in this regard so this type of
geo-data is not available to general public.





On Sun, Apr 13, 2014 at 8:15 PM, Federico Leva (Nemo) nemow...@gmail.comwrote:

 Recent discussion on the topic:
 * http://lists.wikimedia.org/pipermail/analytics/2013-
 August/thread.html#857
 * http://thread.gmane.org/gmane.org.wikimedia.analytics/103

 Nemo

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

The example might not have been the most helpful one. Consider a
handlebars
template like this:
a href={{url}}{{title}}/a

True, much better example to state the point. Now, as I think I  mentioned
earlier there are two cases that need to be treated differently than
anything else: links and translations/localizations.

In this case I wouldn't want the url (or translation) to be plainly parsed.
Rather I would do:

a href={{urlBuilder p1=param1 p2=param2}}{{title}}/a

Where urlBuilder is a user defined function that decides on lawful input
and output scheme.

This would work just the same for translations {{translateGender
maleTranslation femaleTranslation  name=param}} where translateGender is
also defined by us.

But  these are basically the two only schemes you need to treat
differently, the context in these two cases is very precise and thus much
more manageable.

For this you need special and ideally automatic sanitization for href
attributes (and src  style), which is what KnockOff/TAssembly provides.
Sure, that works just as well. But overall is a pretty similar solution to
having a url builder function executed from the template engine with the
drawback that is less performant. I know you guys are set on the DOM based
engine but maybe it is worth thinking how to fit client side translations
on that scheme as translations bring their own escaping problems (if you
have done so please disregard).

My bigger point was to highlite that with a string concatenation engine you
can satisfy security concerns plus have a template engine that performs
really well if you respect the data and markup separation.












On Tue, Apr 1, 2014 at 10:23 PM, Gabriel Wicke gwi...@wikimedia.org wrote:

 On 03/30/2014 02:23 AM, Nuria Ruiz wrote:
  What I am saying is that the parsing and escaping scheme we need is much
  simpler if you disallow the use case of passing the template engine
  something that is not data.
 
  Let me explain as this as it has to do more with correctness that with
  security per se:
  A template engine objective is to separate data from markup. In your
  example you are passing the template 'class=anything' or
  'onclick=something' neither class nor onclick are data.

 The example might not have been the most helpful one. Consider a handlebars
 template like this:

 a href={{url}}{{title}}/a

 Even with double-stashes you'll be in trouble if your url data happens to
 be
 'javascript:alert(cookie)'. For this you need special and ideally automatic
 sanitization for href attributes (and src  style), which is what
 KnockOff/TAssembly provides.

 Gabriel

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

Are you calling handlebars a string concatenation engine?
I think I just invented this string concatenation engine expression but
...yes. Handlebars uses a lexical parser but once templates are compiled
parameter substitution is done concatenating strings. I was referring to
this second step.

Example Template:
h1Handlebars JS Example/h1
script id=some-template type=text/x-handlebars-template table
div
Name {{name}}
/div
/script


Template once compiled:

(function() {
  var template = Handlebars.template, templates = Handlebars.templates =
Handlebars.templates || {};
templates['template_test.html'] = template({compiler:[5,=
2.0.0],main:function(depth0,helpers,partials,data) {
  var helper, functionType=function,
escapeExpression=this.escapeExpression;
  return h1Handlebars JS Example/h1\nscript id=\some-template\
type=\text/x-handlebars-template\ table\ndiv \nName 
+ escapeExpression(((helper = helpers.name || (depth0 
depth0.name)),(typeof
helper === functionType ? helper.call(depth0,
{name:name,hash:{},data:data}) : helper)))
+  \n/div\n/script\n\n;
},useData:true});
})();



On Wed, Apr 2, 2014 at 3:28 PM, C. Scott Ananian canan...@wikimedia.orgwrote:

 Are you calling handlebars a string concatenation engine? In the
 spacebars implementation (and my/gwicke's prototype) it is a structured DOM
 engine.  Don't confuse surface syntax with implementation.
   --scott
 On Apr 2, 2014 7:33 AM, Nuria Ruiz nu...@wikimedia.org wrote:

  The example might not have been the most helpful one. Consider a
  handlebars
  template like this:
  a href={{url}}{{title}}/a
 
  True, much better example to state the point. Now, as I think I
  mentioned
  earlier there are two cases that need to be treated differently than
  anything else: links and translations/localizations.
 
  In this case I wouldn't want the url (or translation) to be plainly
 parsed.
  Rather I would do:
 
  a href={{urlBuilder p1=param1 p2=param2}}{{title}}/a
 
  Where urlBuilder is a user defined function that decides on lawful
 input
  and output scheme.
 
  This would work just the same for translations {{translateGender
  maleTranslation femaleTranslation  name=param}} where translateGender is
  also defined by us.
 
  But  these are basically the two only schemes you need to treat
  differently, the context in these two cases is very precise and thus much
  more manageable.
 
  For this you need special and ideally automatic sanitization for href
  attributes (and src  style), which is what KnockOff/TAssembly
 provides.
  Sure, that works just as well. But overall is a pretty similar solution
 to
  having a url builder function executed from the template engine with the
  drawback that is less performant. I know you guys are set on the DOM
 based
  engine but maybe it is worth thinking how to fit client side translations
  on that scheme as translations bring their own escaping problems (if you
  have done so please disregard).
 
  My bigger point was to highlite that with a string concatenation engine
 you
  can satisfy security concerns plus have a template engine that performs
  really well if you respect the data and markup separation.
 
 
 
 
 
 
 
 
 
 
 
 
  On Tue, Apr 1, 2014 at 10:23 PM, Gabriel Wicke gwi...@wikimedia.org
  wrote:
 
   On 03/30/2014 02:23 AM, Nuria Ruiz wrote:
What I am saying is that the parsing and escaping scheme we need is
  much
simpler if you disallow the use case of passing the template engine
something that is not data.
   
Let me explain as this as it has to do more with correctness that
 with
security per se:
A template engine objective is to separate data from markup. In your
example you are passing the template 'class=anything' or
'onclick=something' neither class nor onclick are data.
  
   The example might not have been the most helpful one. Consider a
  handlebars
   template like this:
  
   a href={{url}}{{title}}/a
  
   Even with double-stashes you'll be in trouble if your url data happens
 to
   be
   'javascript:alert(cookie)'. For this you need special and ideally
  automatic
   sanitization for href attributes (and src  style), which is what
   KnockOff/TAssembly provides.
  
   Gabriel
  
   ___
   Wikitech-l mailing list
   Wikitech-l@lists.wikimedia.org
   https://lists.wikimedia.org/mailman/listinfo/wikitech-l
  
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

 div class={{something}}/div is
vulnerable, if something is set to 1234 onClick=doSomething()

 $html = Html::element( 'div', array( 'class' = $anything ),
$anythingElse
 I see. Sorry but where I disagree is that the quote me this replacement
 is a lawful case for the template engine.

I'm not sure I understand what you're saying here. Do you mean
makesafeString in your example shouldn't quote the text, but should instead
remove space characters?=

What I am saying is that the parsing and escaping scheme we need is much
simpler if you disallow the use case of passing the template engine
something that is not data.

Let me explain as this as it has to do more with correctness that with
security per se:
A template engine objective is to separate data from markup. In your
example you are passing the template 'class=anything' or
'onclick=something' neither class nor onclick are data. Thus what I
am arguing is that the template engine should not support the use case of
add any random attribute or javascript to my html element with the right
set of quotes as a lawful use case. The template engine should not be
expected to parse and insert code and onclick is code.

With a new template engine our main objective should be to separate data
from markup, not supporting an style of coding that includes onClick
attributes mixed with HTML which was prevalent years ago or css classes
mixed with controller code.

On my experience reducing use cases for template engine to just data
handling while having specific functions that deal with links and
translations simplifies the escaping problem greatly as you do not need
context aware escaping. You can js-escape any piece of data sent to the
engine cause you know you do not support the use case of sending javascript
to be inserted.


On Wed, Mar 26, 2014 at 6:41 PM, Chris Steipp cste...@wikimedia.org wrote:

 On Wed, Mar 26, 2014 at 10:30 AM, Nuria Ruiz nu...@wikimedia.org wrote:

  Additionally, how you escape a plain parameter like class vs. an
  href vs. a parameter that is inserted into a url vs. an id attribute are
  all different escaping strategies.
  Urls in the template engine need to be handled on their own, sure. But
 what
  template engine does not work in this fashion? There are three separate
  entities you normally deal with when doing replacement: translations,
  urls and plain attributes.
 
 
  $html = Html::element( 'div', array( 'class' = $anything ),
 $anythingElse
  I see. Sorry but where I disagree is that the quote me this replacement
  is a lawful case for the template engine.


 I'm not sure I understand what you're saying here. Do you mean
 makesafeString in your example shouldn't quote the text, but should instead
 remove space characters?


  The line above is doing a lot
  more than purely templating and on my opinion it does little to separate
  data and markup. Which is the very point of having a template engine.
 
  But if you consider that one a lawful use case, you are right. The
 example
  I provided does not help you.
 
 
  On Wed, Mar 26, 2014 at 6:15 PM, Chris Steipp cste...@wikimedia.org
  wrote:
 
   On Wed, Mar 26, 2014 at 9:44 AM, Daniel Friesen
   dan...@nadir-seen-fire.comwrote:
  
On 2014-03-26, 9:32 AM, Nuria Ruiz wrote:
 The issue is that they apply the same escaping, regardless of the
 html context. So, in Twig and mustache, div
   class={{something}}/div
is
 vulnerable, if something is set to 1234 onClick=doSomething().
 Right, the engine would render:

 div class=1234 onClick=doSomething() /div

 because it only escapes HTML by default.
 Now, note that the problem can be fixed with div
   class={{makeStringSafe
 something}}

 Where makestringSafe is a function defined by us and executed
 there
that
 escapes to our liking.
How does a custom function jammed into the middle of a Mustache
  template
fix the issue when the issue is not that foo={{something}} doesn't
escape, but is that quoting is needed instead of escaping, and
 Mustache
isn't context sensitive so neither Mustache or a custom function know
that foo={{something}} is an attribute value in need of quoting?
   
  
   Exactly. Additionally, how you escape a plain parameter like class vs.
 an
   href vs. a parameter that is inserted into a url vs. an id attribute
 are
   all different escaping strategies. So there would be many different
   makeStringSafe and probably quoteAndMakeStringSafe
 functions,
   and code review would have to make sure the right one was being used in
  the
   right place. Which means someone who is familiar with all of the xss
   techniques would need to code review almost all the templates.
  
   For comparison, using our current html templating (as much as it
 sucks):
  
   $html = Html::element( 'div', array( 'class' = $anything ),
  $anythingElse
   );
  
   The developer doesn't need to have any knowledge of what escaping needs
  to
   apply

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

Sumana,

Sorry for my late reply but since you asked for corrections, here are a
couple.

Mustache.js is a popular modern choice.
Not really, mustache has many lack-offs that prevent it from being a
popular choice, among them the lack of a server side compiler and if/else
constructs. Handlebars is a lot more popular. Also twitters flavor of a
string based engine: http://twitter.github.io/hogan.js/


One approach treats HTML as a string (here's a
bunch of bytes to interpolate). From a security perspective, this is
dangerously easy to have vulnerabilities in, because you just naively
insert strings.
This is not correct. String based systems escape the strings they are
interpolating by default.
Take a look at escaping of what is the most popular string-based engine,
handlebars: https://github.com/wycats/handlebars.js/




On Wed, Mar 19, 2014 at 4:27 AM, Sumana Harihareswara suma...@wikimedia.org
 wrote:

 I'm trying to understand what our current situation is and what our
 choices are around HTML templating systems and MediaWiki, so I'm gonna
 note what I think I understand so far in this mail and then would love
 for people to correct me. TL;DR - did we already consense on a
 templating system and I just missed it?

 Description: An HTML templates system (also known as a templating
 engine) lets you (the programmer) write something that looks more like a
 document than it looks like code, then has hooks/entry points/macro
 substitution points (for user input and whatnot) that then invoke code,
 then emits finished HTML for the browser to render.

 Examples: PHP itself is kinda a templating language. In the PHP world,
 Smarty is a somewhat more mature/old-school choice. Mustache.js is a
 popular modern choice. And in other languages, you'd pick a lot of the
 MVC frameworks that are popular, e.g. Django or Jinja in Python.

 Spectrum of approaches: One approach treats HTML as a string (here's a
 bunch of bytes to interpolate). From a security perspective, this is
 dangerously easy to have vulnerabilities in, because you just naively
 insert strings. Then on the other end of the spectrum, you have code
 that always keeps the document object model (DOM) in memory, so the
 programmer is abstractly manipulating that data model and passing around
 an object. Sure, it spits out HTML in the end, but inherent in the
 method for turning those objects into HTML is a sanitization step, so
 that's inherently more secure. There's some discussion at
 https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing/Templates . I
 presume we want the latter, but that the former model is more performant?

 We talked about this stuff in
 https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-02-21
 and

 https://www.mediawiki.org/wiki/Talk:Architecture_Summit_2014/HTML_templating#Wrap_up:_Next_steps
 . Based on that plus

 https://www.mediawiki.org/wiki/Architecture_Summit_2014/RFC_clusters#HTML_templating
 it seems like we are supposed to get consensus on which system(s) to
 use, and we kind of have four things we could choose:

 * oojs - https://www.mediawiki.org/wiki/OOjs_UI -- could use this
 toolkit with one of the template approaches below, or maybe this is
 enough by itself! Currently used inside VisualEditor and I am not sure
 whether any other MediaWiki extensions or teams are using it? This is a
 DOM-based templating system.

 Template approaches which are competing?:
 * MVC framework - Wikia has written their own templating library that
 Wikia uses (Nirvana). Owen Davis is talking about this tomorrow in the
 RFC review meeting.
 https://www.mediawiki.org/wiki/Requests_for_comment/MVC_framework
 * mustache.js stuff - Ryan Kaldari and Chris Steipp mentioned this I think?
 * Knockout-compatible implementation in Node.js  PHP

 https://www.mediawiki.org/wiki/Requests_for_comment/HTML_templating_library/KnockoutProposal#Longer-term_architecture
 and

 https://www.mediawiki.org/wiki/Requests_for_comment/HTML_templating_library/Knockoff_-_Tassembly
 , being worked on by Gabriel Wicke, Matt Walker, and others. DOM-based.

 There's also an OutputPage refactor suggested in
 https://www.mediawiki.org/wiki/Requests_for_comment/OutputPage_refactor
 that's part of the HTML Templating RFC Cluster

 https://www.mediawiki.org/wiki/Architecture_Summit_2014/RFC_clusters#HTML_templating
 .

 I guess my biggest question right now is whether I have all the big
 moving parts right in my summary above. Thanks.

 --
 Sumana Harihareswara
 Senior Technical Writer
 Wikimedia Foundation

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

The issue is that they apply the same escaping, regardless of the
html context. So, in Twig and mustache, div class={{something}}/div is
vulnerable, if something is set to 1234 onClick=doSomething().

Right, the engine would render:

div class=1234 onClick=doSomething() /div

because it only escapes HTML by default.
Now, note that the problem can be fixed with div class={{makeStringSafe
something}}

Where makestringSafe is a function defined by us and executed there that
escapes to our liking.

This is equivalent to what we would need to do in PHP
server side if we used PHP native templating.
We would need to implement an escaping function like makeStringSafe that
we would execute like:

div class=View::makeStringSafe($this-something)/div

What I wanted to clarify (regarding Sumana's first e-mail) is that when it
comes to security we would need to take the same precautions with a string
based javascript engine than we would do with any PHP engine we choose.
Namely, as you said, spot the lack of makestringSafe via CRs.

To be completely fair, the makeStringSafe could be inserted with a
build/template compilation process and thus we would not need CRs but
rather we could rely on automation.













On Wed, Mar 26, 2014 at 2:43 PM, Chris Steipp cste...@wikimedia.org wrote:

 On Wed, Mar 26, 2014 at 3:21 AM, Nuria Ruiz nu...@wikimedia.org wrote:

  So for string-based systems to be
  as safe as dom ones, we also need a layer of policy and code review that
  we
  might not need with a dom-based system.
  String based template engines (like handlebars) do escape as a default,
 you
  have to use special markup for it not to escape. Can you explain in
 more
  detail what is the security concern with those?
 

 Correct. The issue is that they apply the same escaping, regardless of the
 html context. So, in Twig and mustache, div class={{something}}/div is
 vulnerable, if something is set to 1234 onClick=doSomething(). So
 policy/code review is needed to say that attributes with user-supplied data
 must be quoted in a way compatible with the templating engine (' or  for
 Twig,  for Mustache since Mustache doesn't escape single quotes).


 
 
 
 
 
  On Wed, Mar 19, 2014 at 7:51 PM, Chris Steipp cste...@wikimedia.org
  wrote:
 
   On Tue, Mar 18, 2014 at 8:27 PM, Sumana Harihareswara 
   suma...@wikimedia.org
wrote:
  
I'm trying to understand what our current situation is and what our
choices are around HTML templating systems and MediaWiki, so I'm
 gonna
note what I think I understand so far in this mail and then would
 love
for people to correct me. TL;DR - did we already consense on a
templating system and I just missed it?
   
Description: An HTML templates system (also known as a templating
engine) lets you (the programmer) write something that looks more
 like
  a
document than it looks like code, then has hooks/entry points/macro
substitution points (for user input and whatnot) that then invoke
 code,
then emits finished HTML for the browser to render.
   
Examples: PHP itself is kinda a templating language. In the PHP
 world,
Smarty is a somewhat more mature/old-school choice. Mustache.js is a
popular modern choice. And in other languages, you'd pick a lot of
 the
MVC frameworks that are popular, e.g. Django or Jinja in Python.
   
Spectrum of approaches: One approach treats HTML as a string
 (here's a
bunch of bytes to interpolate). From a security perspective, this is
dangerously easy to have vulnerabilities in, because you just naively
insert strings. Then on the other end of the spectrum, you have code
that always keeps the document object model (DOM) in memory, so the
programmer is abstractly manipulating that data model and passing
  around
an object. Sure, it spits out HTML in the end, but inherent in the
method for turning those objects into HTML is a sanitization step, so
that's inherently more secure. There's some discussion at
https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing/Templates.
  I
presume we want the latter, but that the former model is more
  performant?
   
  
   I don't want to build too much of a straw man against string-based
  systems,
   so it's probably more appropriate to say that the same escaping is
  applied
   to all strings regardless of the html context, or the developer is
   responsible for applying custom escaping. So for string-based systems
 to
  be
   as safe as dom ones, we also need a layer of policy and code review
 that
  we
   might not need with a dom-based system.
  
   Performance of the dom-based systems has turned out to be not that bad,
  but
   performance is a major factor in any engine we go with.
  
  
  
   
We talked about this stuff in
   
  
 
 https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-02-21
and
   
   
  
 
 https://www.mediawiki.org/wiki/Talk:Architecture_Summit_2014/HTML_templating#Wrap_up

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

How does a custom function jammed into the middle of a Mustache template
fix the issue when the issue is not that foo={{something}} doesn't
escape, but is that quoting is needed instead of escaping, and Mustache
isn't context sensitive so neither Mustache or a custom function know
that foo={{something}} is an attribute value in need of quoting?

Sorry but I think you might have missunderstood Chris' example. Attributes
should not need any quoting, that is not a real use case. Place holders are
replaced by attributes that might be extra-escaped but in any case the
template engine should infer anything as to the content being replaced.

The expected outcome after substitution should be: div
class=some-escaped-text /div








On Wed, Mar 26, 2014 at 5:44 PM, Daniel Friesen
dan...@nadir-seen-fire.comwrote:

 On 2014-03-26, 9:32 AM, Nuria Ruiz wrote:
  The issue is that they apply the same escaping, regardless of the
  html context. So, in Twig and mustache, div class={{something}}/div
 is
  vulnerable, if something is set to 1234 onClick=doSomething().
  Right, the engine would render:
 
  div class=1234 onClick=doSomething() /div
 
  because it only escapes HTML by default.
  Now, note that the problem can be fixed with div class={{makeStringSafe
  something}}
 
  Where makestringSafe is a function defined by us and executed there
 that
  escapes to our liking.
 How does a custom function jammed into the middle of a Mustache template
 fix the issue when the issue is not that foo={{something}} doesn't
 escape, but is that quoting is needed instead of escaping, and Mustache
 isn't context sensitive so neither Mustache or a custom function know
 that foo={{something}} is an attribute value in need of quoting?

 ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]



 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating systems MediaWiki - is this summary right?

[Nuria Ruiz]

Additionally, how you escape a plain parameter like class vs. an
href vs. a parameter that is inserted into a url vs. an id attribute are
all different escaping strategies.
Urls in the template engine need to be handled on their own, sure. But what
template engine does not work in this fashion? There are three separate
entities you normally deal with when doing replacement: translations,
urls and plain attributes.


$html = Html::element( 'div', array( 'class' = $anything ), $anythingElse
I see. Sorry but where I disagree is that the quote me this replacement
is a lawful case for the template engine. The line above is doing a lot
more than purely templating and on my opinion it does little to separate
data and markup. Which is the very point of having a template engine.

But if you consider that one a lawful use case, you are right. The example
I provided does not help you.


On Wed, Mar 26, 2014 at 6:15 PM, Chris Steipp cste...@wikimedia.org wrote:

 On Wed, Mar 26, 2014 at 9:44 AM, Daniel Friesen
 dan...@nadir-seen-fire.comwrote:

  On 2014-03-26, 9:32 AM, Nuria Ruiz wrote:
   The issue is that they apply the same escaping, regardless of the
   html context. So, in Twig and mustache, div
 class={{something}}/div
  is
   vulnerable, if something is set to 1234 onClick=doSomething().
   Right, the engine would render:
  
   div class=1234 onClick=doSomething() /div
  
   because it only escapes HTML by default.
   Now, note that the problem can be fixed with div
 class={{makeStringSafe
   something}}
  
   Where makestringSafe is a function defined by us and executed there
  that
   escapes to our liking.
  How does a custom function jammed into the middle of a Mustache template
  fix the issue when the issue is not that foo={{something}} doesn't
  escape, but is that quoting is needed instead of escaping, and Mustache
  isn't context sensitive so neither Mustache or a custom function know
  that foo={{something}} is an attribute value in need of quoting?
 

 Exactly. Additionally, how you escape a plain parameter like class vs. an
 href vs. a parameter that is inserted into a url vs. an id attribute are
 all different escaping strategies. So there would be many different
 makeStringSafe and probably quoteAndMakeStringSafe functions,
 and code review would have to make sure the right one was being used in the
 right place. Which means someone who is familiar with all of the xss
 techniques would need to code review almost all the templates.

 For comparison, using our current html templating (as much as it sucks):

 $html = Html::element( 'div', array( 'class' = $anything ), $anythingElse
 );

 The developer doesn't need to have any knowledge of what escaping needs to
 apply to the class attribute vs the text.



  ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
 
 
 
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l