date:20140711

https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-07-16

Wednesday 16 July at 18:00 UTC, we're going to chat with Yair rand and
Slevinski about their vertical writing support proposal. This will be in
#wikimedia-office on Freenode IRC. Please check it out and comment ahead of
time:
https://www.mediawiki.org/wiki/Requests_for_comment/Vertical_writing_support

Time:
http://www.timeanddate.com/worldclock/fixedtime.html?hour=18&min=00&sec=0&day=16&month=07&year=2014
 11pm in Lahore
 19:00 in Lisbon
 2pm in New York City
 11am in Seattle
  4am Thursday in Melbourne (sorry.)

Sumana Harihareswara
Senior Technical Writer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Fwd: Tech Talk: Hadoop and Beyond. An overview of Analytics infrastructure, Tuesday!

2014-07-11 Thread Rachel Farrand

Hello!

Please join Nuria Ruiz and Andrew Otto next Tuesday, July 15th at 10am SF
time/5pm UTC

for a 30 min tech talk. You can join our hangout or follow along on
youtube:
https://plus.google.com/u/0/b/103470172168784626509/events/c53ho5esd0luccd09a1c30rlrmg
(please note that a link to join the hangout will be posted in the comments
of this event just as it starts).

You can follow ask questions on IRC during the talk in #wikimedia-dev.

If you are not able to follow along live, a video recording will be posted
here
,
to the MediaWiki YouTube channel immediately following the tech talk for
you to view at any time.

More information about the tech talk:

*Hadoop and Beyond. An overview of Analytics infrastructure*In this tech
talk we will be presenting the analytics infrastructure that we have
recently rolled out in production. By now probably everybody knows that
wikimedia hosts an instance of hadoop from which we are going to extract
pageview data in the near future. But .. how exactly does the data get
there?

We will go over the path that webrequest log data takes from varnish to
kafka (a distributed log buffer) to hadoop and the challenges of deploying
this java-based infrastructure in production. We will also talk about how
can we query the data with hive, an SQL-like interface. How can you set up
this stack on vagrant to play with and, last but not least, how we used
hive recently to provide GLAM folks with image view stats:
https://commons.wikimedia.org/wiki/Commons:GLAMwiki_Toolset_Project/NARA_analytics_pilot

Thanks!
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] 'jquery.json' module has been deprecated and will be removed; please use 'json'

The 'jquery.json' module has been deprecated, and we plan to remove it 
in MW 1.25.  You can replace it with the 'json' module.


* In your extension or gadget's ResourceLoader dependencies, change 
'jquery.json' to 'json'

* Use JSON.stringify instead of $.toJSON
* Use JSON.parse instead of $.parseJSON (the latter is actually part of 
jQuery proper, and not being removed currently, but this allows you to 
be consistent and us the standard ECMAScript 5 JSON API for both 
directions).	


There are some less commonly used JSON APIs in jquery.json.  See the 
code 
(https://git.wikimedia.org/blob/mediawiki%2Fcore.git/5ca4bea3d380911c458659e653320497e501d23c/resources%2Fsrc%2Fjquery.json-deprecate.js), 
and feel free to ask.


The benefit is that users with modern browsers (which have JSON 
built-in), will not have to download any JSON implementation (currently, 
they download jquery.json).  Older browsers will still work, but they 
will download the JSON implementation.


Thanks,

Matt Flaschen

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

Also https://bugzilla.wikimedia.org/show_bug.cgi?id=67888 needs to be
fixed so that people have some chance to test if their tools works
with websocket at least few months before switch to it

On Fri, Jul 11, 2014 at 11:14 PM, Petr Bena  wrote:
> BTW do you have any use data or feedback from people who switched from
> IRC to this one? Based on what you decided you are going to phase it
> out? I bet my shoes when you turn of IRC feed, half of wikipedia will
> break and other half will start blaming all devs here on wikitech-l
> for breaking everything.
>
> You really should do some decent research before. I have no problem
> with deploying new technologies, but phasing out old working ones that
> everyone uses is definitely not an improvement to me. Just go to
> en.wikipedia on IRC feed and have a look on number of users. They all
> will get broken once you shut it off
>
> On Fri, Jul 11, 2014 at 11:09 PM, Petr Bena  wrote:
>> right now it seems to me worse in many points:
>>
>> * it requires more traffic (slower and ineffective)
>> * it requires some extra libraries that enlarge dependency tree
>> * protocol itself is extra complicated and unflexible compared to IRC,
>> which can be programmed in few minutes
>>
>> what's the point in replacing one technology with worse one and
>> forcing everyone to use it?
>>
>> On Fri, Jul 11, 2014 at 8:54 PM, Antoine Musso  wrote:
>>> Le 05/05/2014 11:29, Petr Bena a écrit :
 Given the current specifications I can only support this change as
 long as current IRC feed is preserved as IRC is IMHO, as much as evil
 it looks, more suitable for this than WebSockets.

 I am not saying that IRC is suitable for this and I know that people
 really wanted to get rid of it or replace it with something better,
 but I just can't see how is this better.
>>>
>>> IRC will be phased out entirely and related code in MediaWiki entirely
>>> removed if it is not of any other use.
>>>
>>> I am not sure what is the communication / migration plan for RCStream
>>> but you should really start looking at it right now.
>>>
>>>
>>> --
>>> Antoine "hashar" Musso
>>>
>>>
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub


On 07/11/2014 02:54 PM, Alex Monk wrote:

On 11 July 2014 19:49, Antoine Musso  wrote:


If not supported, have a bot detecting such pull requests and autoclose
them with instructions about how to create an account on labs and push a
change.



If I remember correctly, Yuvi had a bot that did something like this. It
broke in some way and no one has fixed it.


Yep, I think this is the best solution.  It automatically uploads a 
change to Gerrit when someone posts a pull request.


I talked to Yuvi on IRC, and he is hoping to make time to fix it next week.

Matt Flaschen


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

BTW do you have any use data or feedback from people who switched from
IRC to this one? Based on what you decided you are going to phase it
out? I bet my shoes when you turn of IRC feed, half of wikipedia will
break and other half will start blaming all devs here on wikitech-l
for breaking everything.

You really should do some decent research before. I have no problem
with deploying new technologies, but phasing out old working ones that
everyone uses is definitely not an improvement to me. Just go to
en.wikipedia on IRC feed and have a look on number of users. They all
will get broken once you shut it off

On Fri, Jul 11, 2014 at 11:09 PM, Petr Bena  wrote:
> right now it seems to me worse in many points:
>
> * it requires more traffic (slower and ineffective)
> * it requires some extra libraries that enlarge dependency tree
> * protocol itself is extra complicated and unflexible compared to IRC,
> which can be programmed in few minutes
>
> what's the point in replacing one technology with worse one and
> forcing everyone to use it?
>
> On Fri, Jul 11, 2014 at 8:54 PM, Antoine Musso  wrote:
>> Le 05/05/2014 11:29, Petr Bena a écrit :
>>> Given the current specifications I can only support this change as
>>> long as current IRC feed is preserved as IRC is IMHO, as much as evil
>>> it looks, more suitable for this than WebSockets.
>>>
>>> I am not saying that IRC is suitable for this and I know that people
>>> really wanted to get rid of it or replace it with something better,
>>> but I just can't see how is this better.
>>
>> IRC will be phased out entirely and related code in MediaWiki entirely
>> removed if it is not of any other use.
>>
>> I am not sure what is the communication / migration plan for RCStream
>> but you should really start looking at it right now.
>>
>>
>> --
>> Antoine "hashar" Musso
>>
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

right now it seems to me worse in many points:

* it requires more traffic (slower and ineffective)
* it requires some extra libraries that enlarge dependency tree
* protocol itself is extra complicated and unflexible compared to IRC,
which can be programmed in few minutes

what's the point in replacing one technology with worse one and
forcing everyone to use it?

On Fri, Jul 11, 2014 at 8:54 PM, Antoine Musso  wrote:
> Le 05/05/2014 11:29, Petr Bena a écrit :
>> Given the current specifications I can only support this change as
>> long as current IRC feed is preserved as IRC is IMHO, as much as evil
>> it looks, more suitable for this than WebSockets.
>>
>> I am not saying that IRC is suitable for this and I know that people
>> really wanted to get rid of it or replace it with something better,
>> but I just can't see how is this better.
>
> IRC will be phased out entirely and related code in MediaWiki entirely
> removed if it is not of any other use.
>
> I am not sure what is the communication / migration plan for RCStream
> but you should really start looking at it right now.
>
>
> --
> Antoine "hashar" Musso
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

I was wrong - the code is merged but we need to have more architectural
discussion before we deploy, so I moved it out of the RfC archive.

Sumana Harihareswara
Senior Technical Writer
Wikimedia Foundation


On Fri, Jul 11, 2014 at 1:18 PM, Sumana Harihareswara  wrote:

> This was merged on May 9th, so I have moved the RfC
> https://www.mediawiki.org/wiki/Requests_for_comment/Publishing_the_RecentChanges_feed
> to the archive - is that right?
>
> Can I understand better where we're using PubSubHubbub vs WebSockets vs
> rcstream?
>
> Sumana Harihareswara
> Senior Technical Writer
> Wikimedia Foundation
>
>
> On Sun, May 4, 2014 at 10:23 PM, Ori Livneh  wrote:
>
>> Hi,
>>
>> Gerrit change Id819246a9 proposes an implementation for a recent changes
>> stream broadcast via socket.io, an abstraction layer over WebSockets that
>> also provides long polling as a fallback for older browsers. Comment on <
>> https://gerrit.wikimedia.org/r/#/c/131040/> or the mailing list.
>>
>> Thanks,
>> Ori
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Bartosz Dziewoński

On Fri, 11 Jul 2014 21:35:56 +0200, Merlijn van Deen  
 wrote:



I have plans to extend the uploader to allow simpele github uploads (just
like the bugzilla option), but I have not had time to work on this yet.  
If

anyone feels like working on it: see
https://github.com/valhallasw/gerrit-patch-uploader/blob/master/patchuploader.py#L92
for the existing download-a-patch-from-bugzilla implementation. I'd  
happily

merge a patch for a one-click github uploader.


For anyone planning to work on this, a tip: you can add ".patch" or  
".diff" to a GitHub pull request URL to get a  
Gerrit-Patch-Uploader-compatible patch, e.g.  
https://github.com/wikimedia/mediawiki-core/pull/19.diff


--
Matma Rex

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

On Fri, Jul 11, 2014 at 2:49 PM, Antoine Musso  wrote:

> Le 11/07/2014 20:34, Matthew Flaschen a écrit :
> >
> >> According to <
> >> https://github.com/wikimedia/mediawiki-core/settings/hooks> (may not be
> >> visible to non-Wikimedians, sorry), the WebHook receiver <
> >> http://tools.wmflabs.org/suchaserver/cgi-bin/receiver.py> is defunct.
> >> Anyone know the story there?
> >
> > It would be good to get this bot (which uploads things to Gerrit) up and
> > running again.  I'm not sure what the current status is.
>
> There is a manual tool at:
>
>   https://tools.wmflabs.org/gerrit-patch-uploader/
>   https://gerrit.wikimedia.org/r/#/q/owner:gerritpatchuploader,n,z
>
>
> Krenair manually warned some folks on Github about having to send the
> patches to Gerrit.
>
> What would be nice: disable pull requests on Github.
>
> If not supported, have a bot detecting such pull requests and autoclose
> them with instructions about how to create an account on labs and push a
> change.
>
>
I personally think that fixing up Merlijn's Gerrit patch uploader would
lead to a far superior developer experience.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Merlijn van Deen

On 11 July 2014 21:49, Antoine Musso  wrote:

> There is a manual tool at:
>
>   https://tools.wmflabs.org/gerrit-patch-uploader/
>   https://gerrit.wikimedia.org/r/#/q/owner:gerritpatchuploader,n,z

I have plans to extend the uploader to allow simpele github uploads (just
like the bugzilla option), but I have not had time to work on this yet. If
anyone feels like working on it: see
https://github.com/valhallasw/gerrit-patch-uploader/blob/master/patchuploader.py#L92
for the existing download-a-patch-from-bugzilla implementation. I'd happily
merge a patch for a one-click github uploader.

Merlijn
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Chad

On Fri, Jul 11, 2014 at 11:49 AM, Antoine Musso  wrote:

> What would be nice: disable pull requests on Github.
>
> If not supported, have a bot detecting such pull requests and autoclose
> them with instructions about how to create an account on labs and push a
> change.
>
>
Not possible to disable.

-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

2014-07-11 Thread Antoine Musso

Le 05/05/2014 11:29, Petr Bena a écrit :
> Given the current specifications I can only support this change as
> long as current IRC feed is preserved as IRC is IMHO, as much as evil
> it looks, more suitable for this than WebSockets.
> 
> I am not saying that IRC is suitable for this and I know that people
> really wanted to get rid of it or replace it with something better,
> but I just can't see how is this better.

IRC will be phased out entirely and related code in MediaWiki entirely
removed if it is not of any other use.

I am not sure what is the communication / migration plan for RCStream
but you should really start looking at it right now.


-- 
Antoine "hashar" Musso


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Alex Monk

On 11 July 2014 19:49, Antoine Musso  wrote:

> If not supported, have a bot detecting such pull requests and autoclose
> them with instructions about how to create an account on labs and push a
> change.
>

If I remember correctly, Yuvi had a bot that did something like this. It
broke in some way and no one has fixed it.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Antoine Musso

Le 11/07/2014 20:34, Matthew Flaschen a écrit :
> 
>> According to <
>> https://github.com/wikimedia/mediawiki-core/settings/hooks> (may not be
>> visible to non-Wikimedians, sorry), the WebHook receiver <
>> http://tools.wmflabs.org/suchaserver/cgi-bin/receiver.py> is defunct.
>> Anyone know the story there?
> 
> It would be good to get this bot (which uploads things to Gerrit) up and
> running again.  I'm not sure what the current status is.

There is a manual tool at:

  https://tools.wmflabs.org/gerrit-patch-uploader/
  https://gerrit.wikimedia.org/r/#/q/owner:gerritpatchuploader,n,z


Krenair manually warned some folks on Github about having to send the
patches to Gerrit.

What would be nice: disable pull requests on Github.

If not supported, have a bot detecting such pull requests and autoclose
them with instructions about how to create an account on labs and push a
change.


-- 
Antoine "hashar" Musso


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub

2014-07-11 Thread Chad

On Fri, Jul 11, 2014 at 11:34 AM, Matthew Flaschen 
wrote:

> On 04/18/2014 04:21 AM, Ori Livneh wrote:
>
>> Is there a way to accept pull-requests from GitHub?
>>
>
> I don't think we can merge directly there.  The canonical repo is
> git.wikimedia.org, and I think merging in GitHub would create an
> inconsistent state.
>
>
The canonical repo source is gerrit.wikimedia.org.

git.wikimedia.org is a mirror.

But otherwise yes, you're right.

-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minding pull-requests on GitHub


On 04/18/2014 04:21 AM, Ori Livneh wrote:

Is there a way to accept pull-requests from GitHub?


I don't think we can merge directly there.  The canonical repo is 
git.wikimedia.org, and I think merging in GitHub would create an 
inconsistent state.



According to <
https://github.com/wikimedia/mediawiki-core/settings/hooks> (may not be
visible to non-Wikimedians, sorry), the WebHook receiver <
http://tools.wmflabs.org/suchaserver/cgi-bin/receiver.py> is defunct.
Anyone know the story there?


It would be good to get this bot (which uploads things to Gerrit) up and 
running again.  I'm not sure what the current status is.



It'd be good if some additional people were watching (that is, receiving
notifications for) .


I've watched it.

Matt Flaschen

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

it may surprise you but there are other languages than python and in
these, things like this aren't that simple...

On Sun, May 11, 2014 at 1:14 AM, Alex Monk  wrote:
> I've fiddled around with this a bit:
>
> On my Ubuntu system, I simply got a library: sudo pip install
> socketio-client
> Then made this Python script:
>
> ---
> from socketIO_client import SocketIO, BaseNamespace
> class RecentChangeNamespace(BaseNamespace):
> def on_connect(self):
> self.emit('subscribe', ['enwiki'])
> def on_change(self, data):
> print(data)
>
> socketIO = SocketIO('rcstream.wmflabs.org') # This host seems to be sending
> the same example entries at the moment, obviously we should get an actual
> RC stream from it or a production host later
> socketIO.define(RecentChangeNamespace, '/rc')
> socketIO.wait()
> ---
> And it prints entries to console like this (Python just prints out objects
> in a way that kind of looks like JSON):
> {u'comment': u'', u'wiki': u'enwiki', u'type': 0, u'title': u'Main Page',
> u'timestamp': 1398993841, u'server_script_path': u'/w', u'namespace': 0,
> u'server_url': u'http://en.wikipedia.beta.wmflabs.org', u'length': {u'new':
> 585, u'old': 580}, u'user': u'Alice', u'patrolled': True, u'bot': False,
> u'id': 9, u'minor': True, u'revision': {u'new': 9, u'old': 8}}
>
> It's so much nicer. Sorry Petr, I think this is much more suitable than the
> mess that is connecting to IRC and parsing everything. I definitely want to
> see this replace the IRC feed entirely (with the obvious reasonably long
> deprecation period).
>
>
> Alex
>
>
> On 5 May 2014 10:29, Petr Bena  wrote:
>
>> Given the current specifications I can only support this change as
>> long as current IRC feed is preserved as IRC is IMHO, as much as evil
>> it looks, more suitable for this than WebSockets.
>>
>> I am not saying that IRC is suitable for this and I know that people
>> really wanted to get rid of it or replace it with something better,
>> but I just can't see how is this better.
>>
>> On Mon, May 5, 2014 at 10:37 AM, Daniel Kinzler 
>> wrote:
>> > Am 05.05.2014 07:20, schrieb Jeremy Baron:
>> >> On May 4, 2014 10:24 PM, "Ori Livneh"  wrote:
>> >>> an implementation for a recent changes
>> >>> stream broadcast via socket.io, an abstraction layer over WebSockets
>> that
>> >>> also provides long polling as a fallback for older browsers.
>> >
>> > [...]
>> >
>> >> How could this work overlap with adding pubsubhubbub support to existing
>> >> web RC feeds? (i.e. atom/rss. or for that matter even individual page
>> >> history feeds or related changes feeds)
>> >>
>> >> The only pubsubhubbub bugs I see atm are
>> >> https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=38970%2C30245
>> >
>> > There is a Pubsubhubbub implementation in the pipeline, see
>> > .
>> It's
>> > pretty simple and painless. We plan to have this deployed experimentally
>> for
>> > wikidata soon, but there is no reason not to roll it out globally.
>> >
>> > This implementation uses the job queue - which in production means
>> redis, but
>> > it's pretty generic.
>> >
>> > As to an RC *stream*: Pubsubhubbub is not really suitable for this,
>> since it
>> > requires the subscriber to run a public web server. It's really a
>> > server-to-server protocol. I'm not too sure about web sockets for this
>> either,
>> > because the intended recipient is usually not a web browser. But if it
>> works,
>> > I'd be happy anyway, the UDP+IRC solution sucks.
>> >
>> > Some years ago, I started to implement an XMPP based RC stream, see
>> > . Have a look and steal
>> some
>> > ideas :)
>> >
>> > -- daniel
>> >
>> >
>> >
>> > ___
>> > Wikitech-l mailing list
>> > Wikitech-l@lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses


On 07/11/2014 11:45 AM, Brion Vibber wrote:

As I recall, UseModWiki (the perl-based wiki software we used before
switching to a custom solution which evolved into MediaWiki) obscured the
last octet of the IP address, which still left you with enough information
in most cases to track down an ISP or school/business/govt institution. I
think UseMod also exposed the IP addresses of logged-in users, but the way
logins worked were very different and it was possible to set your name to
someone else's name or some such oddities...


Yeah, the main benefit to the current setup (which probably doesn't 
really require the last octet in most cases) is detecting casual abuse, 
which includes (but is not limited to) both blatant vandalism and 
conflict of interest edits.  (People have lunch breaks, and I don't 
claim every edit from a organizational IP is a conflict of interest, but 
many true COI edits have been caught this way).


If we look into something like proto-accounts or hashing or such, it 
would be good to try to maintain this benefit (do the lookup on the 
server, and expose who the IP block belongs to?), but I don't know if 
it's possible to have it both ways.


Matt Flaschen


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upgrading jquery cookie

On 07/11/2014 06:12 AM, Thomas Mulhall wrote:

Hi we are upgrading jquery cookie from an early alpha version of 1.1
to 1.2. Please start upgrading your code to be compatible with jquery
cookie 1.2. There is just one deprecations to notice and that is
$.cookie('foo', null) is now deprecated. And replace it with
Adding $.removeCookie('foo') for deleting a cookie. We are
slowly upgrading to version 1.4.1 but one step at a time because it
is. And please also follow this change log
github.com/carhartl/jquery-cookie/blob/master/CHANGELOG.md so that we
can upgrade to 1.4.1.

This refers to two changes:

* https://gerrit.wikimedia.org/r/#/c/145000/ - This will probably be
merged shortly. The only breaking change mentioned is that $.cookie(
'key', undefined ) no longer works. The documented
(https://github.com/carhartl/jquery-cookie/blob/faa09dc38bd3c791212e8fca67ee661af55fa530/README.md)
way to do this for the version we use is $.cookie( 'key', null )

There should also be an update to mw.cookie, either in 145000, or a
follow-up chained on that.

After those change(s) are merged) (upgrading to 1.2, and updating
mw.cookie to use removeCookie), code using jQuery Cookie should either:
** Update to mw.cookie (it is possible to use unprefixed keys if needed
for backwards-copmatibility reasons)

** Or if still using jQuery Cookie directly, use removeCookie.

* https://gerrit.wikimedia.org/r/#/c/139686/ - Upgrades to jQuery Cookie
1.4.1. Not possible until all code using $.cookie( 'key', null ) has
been migrated.

Matt Flaschen

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] recent changes stream

This was merged on May 9th, so I have moved the RfC
https://www.mediawiki.org/wiki/Requests_for_comment/Publishing_the_RecentChanges_feed
to the archive - is that right?

Can I understand better where we're using PubSubHubbub vs WebSockets vs
rcstream?

Sumana Harihareswara
Senior Technical Writer
Wikimedia Foundation

On Sun, May 4, 2014 at 10:23 PM, Ori Livneh  wrote:

> Hi,
>
> Gerrit change Id819246a9 proposes an implementation for a recent changes
> stream broadcast via socket.io, an abstraction layer over WebSockets that
> also provides long polling as a fallback for older browsers. Comment on <
> https://gerrit.wikimedia.org/r/#/c/131040/> or the mailing list.
>
> Thanks,
> Ori
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Happy Melon

On 11 July 2014 17:10, Gilles Dubuc  wrote:

>
> Maybe it's a cookie-based approach you had in mind? Where we automatically
> create an account tied to the user agent. That would mitigate the issue of
> converting a pseudo-account that might have been shared between several
> people to a proper account, but not completely get rid of it.
>

I'd have thought the chain of events "go to a library computer, do some
edits, decide to upgrade to a real account, do so, realise you've
inadvertently swept up all the unsalubrious penis vandalism that has been
made on that computer previously" would be unacceptably common.

--HM
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

> I kinda like the idea of an
> anonymous-but-consistent "proto-account" that can be transformed into a
> named login if desired, but it needs to be thought out in more detail to
> resolve potential difficulties.


One could automatically create a pseudo-account ("Anonymous #12345") upon
first edit. And that account would always be authenticated automaticallly
upon future edits coming from the same IP address. I don't think it should
be allowed to turn those pseudo-accounts into proper accounts, though,
they'd be marked as anonymous pseudo-accounts forever. Otherwise having a
way to upgrade to a proper account while conserving edits which were
potentially written by other people could get hairy, especially from a
legal standpoint.

Maybe it's a cookie-based approach you had in mind? Where we automatically
create an account tied to the user agent. That would mitigate the issue of
converting a pseudo-account that might have been shared between several
people to a proper account, but not completely get rid of it.


On Fri, Jul 11, 2014 at 11:45 AM, Brion Vibber 
wrote:

> On Friday, July 11, 2014, Risker  wrote:
>
> > This is one of those perennial proposals that never quite seems to take
> > off; I can remember having some version of this discussion back in 2008,
> > and I know that some of our earliest edits show a partially obscured IP
> > address, not the whole thing. It might require Brion or Tim or someone
> else
> > of that length of experience to explain the original thinking.
>
>
> As I recall, UseModWiki (the perl-based wiki software we used before
> switching to a custom solution which evolved into MediaWiki) obscured the
> last octet of the IP address, which still left you with enough information
> in most cases to track down an ISP or school/business/govt institution. I
> think UseMod also exposed the IP addresses of logged-in users, but the way
> logins worked were very different and it was possible to set your name to
> someone else's name or some such oddities...
>
> I'm not sure offhand if there was explicit discussion of switching to not
> obscuring the last octet in the PHP software/nascent MediaWiki... But this
> was back in 2001 when the internet was a little younger and everybody was
> spewing their IP addresses all over their email and newsgroup posts too.
> Folks are a lot more paranoid about that today.
>
>
> In general I favor migrating away from publicly exposing IP addresses, but
> not sure to what exactly would be best... I kinda like the idea of an
> anonymous-but-consistent "proto-account" that can be transformed into a
> named login if desired, but it needs to be thought out in more detail to
> resolve potential difficulties.
>
> -- brion
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikitech-ambassadors] Deprecating print-on-demand functionality

2014-07-11 Thread Erik Moeller

On Fri, Jul 11, 2014 at 8:45 AM, Luca Martinelli
 wrote:
> so the Book Creator will still be active, maybe under another name,
> maybe with another engine, but still active?

Same name and functionality, just the "Order a printed book" feature
will disappear.

Erik
-- 
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Brion Vibber

On Friday, July 11, 2014, Risker  wrote:

> This is one of those perennial proposals that never quite seems to take
> off; I can remember having some version of this discussion back in 2008,
> and I know that some of our earliest edits show a partially obscured IP
> address, not the whole thing. It might require Brion or Tim or someone else
> of that length of experience to explain the original thinking.

As I recall, UseModWiki (the perl-based wiki software we used before
switching to a custom solution which evolved into MediaWiki) obscured the
last octet of the IP address, which still left you with enough information
in most cases to track down an ISP or school/business/govt institution. I
think UseMod also exposed the IP addresses of logged-in users, but the way
logins worked were very different and it was possible to set your name to
someone else's name or some such oddities...

I'm not sure offhand if there was explicit discussion of switching to not
obscuring the last octet in the PHP software/nascent MediaWiki... But this
was back in 2001 when the internet was a little younger and everybody was
spewing their IP addresses all over their email and newsgroup posts too.
Folks are a lot more paranoid about that today.

In general I favor migrating away from publicly exposing IP addresses, but
not sure to what exactly would be best... I kinda like the idea of an
anonymous-but-consistent "proto-account" that can be transformed into a
named login if desired, but it needs to be thought out in more detail to
resolve potential difficulties.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Chris Steipp

On Friday, July 11, 2014, Daniel Kinzler  wrote:

> Am 11.07.2014 17:19, schrieb Tyler Romeo:
> > Most likely, we would encrypt the IP with AES or something using a
> > configuration-based secret key. That way checkusers can still reverse the
> > hash back into normal IP addresses without having to store the mapping
> in the
> > database.
>
> There are two problems with this, I think.
>
> 1) No forward secrecy. If that key is ever leaked, all IPs become "plain".
> And
> it will be, sooner or later. This would probably not be obvious, so this
> feature
> would instill a false sense of security.
>

This is probably the biggest issue. Even if we hmac it, it's trivial to
brute force the entire ipv4 (and with intelligent assumptions about
generation, most of the ipv6) range in seconds, if the key was ever known.

>
> 2) No range blocks. It's often quite useful to be able to block a range of
> IPs.
> This is an important tool in the fight against spammers, taking it away
> would be
> a problem.
>

Range blocks, I imagine, would continue working the same way they do.
Someone would have to identify the correct range (which is very difficult
when administrators can't see IP's), but on submission, we have the IP
address to check against the blocks. (Unless someone proposes to store
block ranges as hashes, that would definitely get rid of range blocks).

>
> -- daniel
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Daniel Kinzler

Am 11.07.2014 17:19, schrieb Tyler Romeo:
> Most likely, we would encrypt the IP with AES or something using a
> configuration-based secret key. That way checkusers can still reverse the
> hash back into normal IP addresses without having to store the mapping in the
> database.

There are two problems with this, I think.

1) No forward secrecy. If that key is ever leaked, all IPs become "plain". And
it will be, sooner or later. This would probably not be obvious, so this feature
would instill a false sense of security.

2) No range blocks. It's often quite useful to be able to block a range of IPs.
This is an important tool in the fight against spammers, taking it away would be
a problem.

-- daniel

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Ole Palnatoke Andersen

To my knowledge, there are currently six of these Twitter bots
(Canada, Denmark, France, Sweden, UK, US). I have collected them in a
Twitter list: https://twitter.com/palnatoke/lists/wikiedit

Please speak up if you notice more, so I can include them in the list, too.


Regards,
Ole

On Fri, Jul 11, 2014 at 3:34 PM, Gilles Dubuc  wrote:
> This interesting bot showed up on hackernews today:
> https://news.ycombinator.com/item?id=8018284
>
> While in this instance the access to anonymous' editors IP addresses is
> definitely useful in terms of identifying edits with probable conflict of
> interest, it makes me wonder what the history is behind the fact that
> anonymous editors are identified by their IP addresses on WMF-hosted wikis.
>
> IP addresses are closely guarded for registered users, why wouldn't
> anonymous users be identified by a hash of their IP address in order to
> protect their privacy as well? The exact same functionality of being able
> to see all edits by a given anonymous IP would still exist, the IP itself
> just wouldn't be publicly available, protected with the same access rights
> as registered users'.
>
> The "use case" that makes me think of that is someone living in a
> totalitarian regime making a sensitive edit and forgetting that they're
> logged out. Or just being unaware that being anonymous on the wiki doesn't
> mean that their local authorities can figure out who they are based on IP
> address and time. Understanding that they're somewhat protected when logged
> in and not when logged out requires a certain level of technical
> understanding. The easy way out of this argument is to state that these
> users should be using Tor or something similar. But I still wonder why we
> have this double standard of protecting registered users' privacy in
> regards to IP addresses and not applying the same for anonymous users, when
> simple hashing would do the job.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
http://palnatoke.org * @palnatoke * +4522934588

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Tyler Romeo

As a quick implementation note, we would not be using a hash for the IP address.

Most likely, we would encrypt the IP with AES or something using a 
configuration-based secret key. That way checkusers can still reverse the hash 
back into normal IP addresses without having to store the mapping in the 
database.

-- 
Tyler Romeo
0x405D34A7C86B42DF

From: Gilles Dubuc 
Reply: Wikimedia developers >
Date: July 11, 2014 at 10:59:55
To: Wikimedia developers >
Subject:  Re: [Wikitech-l] Anonymous editors & IP addresses  

With hashing, a given IP would always give the same hash. So this
uniqueness property would remain for people with stable IPs.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

> Even in this day and age, there are plenty of people with stable IPs
>

With hashing, a given IP would always give the same hash. So this
uniqueness property would remain for people with stable IPs.


On Fri, Jul 11, 2014 at 10:55 AM, Risker  wrote:

> This is one of those perennial proposals that never quite seems to take
> off; I can remember having some version of this discussion back in 2008,
> and I know that some of our earliest edits show a partially obscured IP
> address, not the whole thing. It might require Brion or Tim or someone else
> of that length of experience to explain the original thinking.
>
> Some of the "pros" of keeping the IP address as the "username" for
> unregistered users:
>
>- Even in this day and age, there are plenty of people with stable IPs;
>they choose to edit as unregistered users for philosophical reasons, and
>their IP's edit history is essentially their own editing history
>- Especially on smaller projects (but also big ones), range blocks are
>usually calculated and applied by administrators, not
> checkusers/stewards.
>
>
> Some of the "cons" of publishing the IP address as the username:
>
>- Privacy - IPv6 addresses in particular are including more and more
>very specific information that could be used to link RealLife Name with
> the
>edits. (My own ISP now gives enough information in many cases to narrow
>geolocation down to a one-block radius - a big change from 2 years ago
> when
>geolocation was about an 800 mile radius.)
>- Privacy - more and more jurisdictions consider a person's IP address
>to be "private" information.  Our page histories could be considered one
>gigantic privacy violation.
>- Increasingly dynamic IP addresses, often rotating within very large
>ranges that no longer link with any certainty to geolocation
>- Freaked out new users who didn't really get that their IP address was
>going to be very publicly displayed.
>
>
> I'm pretty sure there are a whole pile more pros and cons that we can pull
> out of the archives from various mailing lists, and I know that there have
> periodically been discussions amongst developers and the rest of the
> engineering team to try to come up with a "better way" - but like many
> other interesting, good and even potentially necessary ideas, it's never
> made it to the top of the priority heap.
>
> Putting on my checkuser hat for just a minute...it's essential information
> for having any chance at all of identifying multiple accounts or pattern
> editing; however, the tables used by checkusers are non-public so
> Checkusers continuing to have access to IP data should not be an issue.
>
> Risker/Anne
>
>
> On 11 July 2014 10:25, Tyler Romeo  wrote:
>
> > I agree that it’s a double standard, but looking at the bright side, it
> > becomes a big encouragement to anonymous users to register and log in.
> The
> > Account Creation Experience Team (or whoever the hell is in charge of
> that)
> > can correct me, but I would imagine that we would see a big drop in
> > registered accounts if IPs were hashed.
> >
> > Also, it’d be really annoying to have hashes as usernames, so we’d have
> to
> > think of an alternative scheme that makes things more readable.
> > --
> > Tyler Romeo
> > 0x405D34A7C86B42DF
> >
> > From: Gilles Dubuc 
> > Reply: Wikimedia developers >
> > Date: July 11, 2014 at 9:34:18
> > To: Wikimedia developers >
> > Subject:  [Wikitech-l] Anonymous editors & IP addresses
> >
> > This interesting bot showed up on hackernews today:
> > https://news.ycombinator.com/item?id=8018284
> >
> > While in this instance the access to anonymous' editors IP addresses is
> > definitely useful in terms of identifying edits with probable conflict of
> > interest, it makes me wonder what the history is behind the fact that
> > anonymous editors are identified by their IP addresses on WMF-hosted
> wikis.
> >
> > IP addresses are closely guarded for registered users, why wouldn't
> > anonymous users be identified by a hash of their IP address in order to
> > protect their privacy as well? The exact same functionality of being able
> > to see all edits by a given anonymous IP would still exist, the IP itself
> > just wouldn't be publicly available, protected with the same access
> rights
> > as registered users'.
> >
> > The "use case" that makes me think of that is someone living in a
> > totalitarian regime making a sensitive edit and forgetting that they're
> > logged out. Or just being unaware that being anonymous on the wiki
> doesn't
> > mean that their local authorities can figure out who they are based on IP
> > address and time. Understanding that they're somewhat protected when
> logged
> > in and not when logged out requires a certain level of technical
> > understanding. The easy way out of this argument is to state that these
> > users should be using Tor or something similar. But I still wonder why we
> > have this double standard of protect

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Risker

This is one of those perennial proposals that never quite seems to take
off; I can remember having some version of this discussion back in 2008,
and I know that some of our earliest edits show a partially obscured IP
address, not the whole thing. It might require Brion or Tim or someone else
of that length of experience to explain the original thinking.

Some of the "pros" of keeping the IP address as the "username" for
unregistered users:

   - Even in this day and age, there are plenty of people with stable IPs;
   they choose to edit as unregistered users for philosophical reasons, and
   their IP's edit history is essentially their own editing history
   - Especially on smaller projects (but also big ones), range blocks are
   usually calculated and applied by administrators, not checkusers/stewards.


Some of the "cons" of publishing the IP address as the username:

   - Privacy - IPv6 addresses in particular are including more and more
   very specific information that could be used to link RealLife Name with the
   edits. (My own ISP now gives enough information in many cases to narrow
   geolocation down to a one-block radius - a big change from 2 years ago when
   geolocation was about an 800 mile radius.)
   - Privacy - more and more jurisdictions consider a person's IP address
   to be "private" information.  Our page histories could be considered one
   gigantic privacy violation.
   - Increasingly dynamic IP addresses, often rotating within very large
   ranges that no longer link with any certainty to geolocation
   - Freaked out new users who didn't really get that their IP address was
   going to be very publicly displayed.


I'm pretty sure there are a whole pile more pros and cons that we can pull
out of the archives from various mailing lists, and I know that there have
periodically been discussions amongst developers and the rest of the
engineering team to try to come up with a "better way" - but like many
other interesting, good and even potentially necessary ideas, it's never
made it to the top of the priority heap.

Putting on my checkuser hat for just a minute...it's essential information
for having any chance at all of identifying multiple accounts or pattern
editing; however, the tables used by checkusers are non-public so
Checkusers continuing to have access to IP data should not be an issue.

Risker/Anne


On 11 July 2014 10:25, Tyler Romeo  wrote:

> I agree that it’s a double standard, but looking at the bright side, it
> becomes a big encouragement to anonymous users to register and log in. The
> Account Creation Experience Team (or whoever the hell is in charge of that)
> can correct me, but I would imagine that we would see a big drop in
> registered accounts if IPs were hashed.
>
> Also, it’d be really annoying to have hashes as usernames, so we’d have to
> think of an alternative scheme that makes things more readable.
> --
> Tyler Romeo
> 0x405D34A7C86B42DF
>
> From: Gilles Dubuc 
> Reply: Wikimedia developers >
> Date: July 11, 2014 at 9:34:18
> To: Wikimedia developers >
> Subject:  [Wikitech-l] Anonymous editors & IP addresses
>
> This interesting bot showed up on hackernews today:
> https://news.ycombinator.com/item?id=8018284
>
> While in this instance the access to anonymous' editors IP addresses is
> definitely useful in terms of identifying edits with probable conflict of
> interest, it makes me wonder what the history is behind the fact that
> anonymous editors are identified by their IP addresses on WMF-hosted wikis.
>
> IP addresses are closely guarded for registered users, why wouldn't
> anonymous users be identified by a hash of their IP address in order to
> protect their privacy as well? The exact same functionality of being able
> to see all edits by a given anonymous IP would still exist, the IP itself
> just wouldn't be publicly available, protected with the same access rights
> as registered users'.
>
> The "use case" that makes me think of that is someone living in a
> totalitarian regime making a sensitive edit and forgetting that they're
> logged out. Or just being unaware that being anonymous on the wiki doesn't
> mean that their local authorities can figure out who they are based on IP
> address and time. Understanding that they're somewhat protected when logged
> in and not when logged out requires a certain level of technical
> understanding. The easy way out of this argument is to state that these
> users should be using Tor or something similar. But I still wonder why we
> have this double standard of protecting registered users' privacy in
> regards to IP addresses and not applying the same for anonymous users, when
> simple hashing would do the job.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wik

Re: [Wikitech-l] Anonymous editors & IP addresses

>
> I would imagine that we would see a big drop in registered accounts if IPs
> were hashed.
>

Why? Most casual web users don't even know what an IP address is, let alone
what their own address is. In fact the evolution of browsers tends to even
hide the URL. This is the sort of technical information that an
ever-shrinking portion of web users know about these days.

an alternative scheme that makes things more readable
>

A hash can take many forms. In fact it could be formatted just like an IP
address. Even if the hash format mixes letters and numbers, as long as the
length is similar, I don't see how IP addresses are superior in terms of
readability.


On Fri, Jul 11, 2014 at 10:25 AM, Tyler Romeo  wrote:

> I agree that it’s a double standard, but looking at the bright side, it
> becomes a big encouragement to anonymous users to register and log in. The
> Account Creation Experience Team (or whoever the hell is in charge of that)
> can correct me, but I would imagine that we would see a big drop in
> registered accounts if IPs were hashed.
>
> Also, it’d be really annoying to have hashes as usernames, so we’d have to
> think of an alternative scheme that makes things more readable.
> --
> Tyler Romeo
> 0x405D34A7C86B42DF
>
> From: Gilles Dubuc  
> Reply: Wikimedia developers >
> 
> Date: July 11, 2014 at 9:34:18
> To: Wikimedia developers >
> 
> Subject:  [Wikitech-l] Anonymous editors & IP addresses
>
> This interesting bot showed up on hackernews today:
> https://news.ycombinator.com/item?id=8018284
>
> While in this instance the access to anonymous' editors IP addresses is
> definitely useful in terms of identifying edits with probable conflict of
> interest, it makes me wonder what the history is behind the fact that
> anonymous editors are identified by their IP addresses on WMF-hosted wikis.
>
> IP addresses are closely guarded for registered users, why wouldn't
> anonymous users be identified by a hash of their IP address in order to
> protect their privacy as well? The exact same functionality of being able
> to see all edits by a given anonymous IP would still exist, the IP itself
> just wouldn't be publicly available, protected with the same access rights
> as registered users'.
>
> The "use case" that makes me think of that is someone living in a
> totalitarian regime making a sensitive edit and forgetting that they're
> logged out. Or just being unaware that being anonymous on the wiki doesn't
> mean that their local authorities can figure out who they are based on IP
> address and time. Understanding that they're somewhat protected when logged
> in and not when logged out requires a certain level of technical
> understanding. The easy way out of this argument is to state that these
> users should be using Tor or something similar. But I still wonder why we
> have this double standard of protecting registered users' privacy in
> regards to IP addresses and not applying the same for anonymous users, when
> simple hashing would do the job.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] User agent policy for bots

2014-07-11 Thread Marc A. Pelletier

On 07/11/2014 10:07 AM, Jeremy Baron wrote:
> I can recall more than a few past conversations about this with mzmcbride,
> Tim Starling and others.

Keep in mind I've only been around for ~18 months, so I am going to be
unaware of some previous discussion on the subject.

But yeah, /clearly/ the more specific an UA is the more precise any
intervention can be.  I'm not sure how reasonable it is to extrapolate
from that to "must have email/irc nick/etc".  You want your UA to be as
unique as possible, certainly, and the more info you give the more
likely it is that we are able to talk to you before we take drastic
measures.

-- Marc

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Anonymous editors & IP addresses

2014-07-11 Thread Tyler Romeo

I agree that it’s a double standard, but looking at the bright side, it becomes 
a big encouragement to anonymous users to register and log in. The Account 
Creation Experience Team (or whoever the hell is in charge of that) can correct 
me, but I would imagine that we would see a big drop in registered accounts if 
IPs were hashed.

Also, it’d be really annoying to have hashes as usernames, so we’d have to 
think of an alternative scheme that makes things more readable.
-- 
Tyler Romeo
0x405D34A7C86B42DF

From: Gilles Dubuc 
Reply: Wikimedia developers >
Date: July 11, 2014 at 9:34:18
To: Wikimedia developers >
Subject:  [Wikitech-l] Anonymous editors & IP addresses  

This interesting bot showed up on hackernews today:
https://news.ycombinator.com/item?id=8018284

While in this instance the access to anonymous' editors IP addresses is
definitely useful in terms of identifying edits with probable conflict of
interest, it makes me wonder what the history is behind the fact that
anonymous editors are identified by their IP addresses on WMF-hosted wikis.

IP addresses are closely guarded for registered users, why wouldn't
anonymous users be identified by a hash of their IP address in order to
protect their privacy as well? The exact same functionality of being able
to see all edits by a given anonymous IP would still exist, the IP itself
just wouldn't be publicly available, protected with the same access rights
as registered users'.

The "use case" that makes me think of that is someone living in a
totalitarian regime making a sensitive edit and forgetting that they're
logged out. Or just being unaware that being anonymous on the wiki doesn't
mean that their local authorities can figure out who they are based on IP
address and time. Understanding that they're somewhat protected when logged
in and not when logged out requires a certain level of technical
understanding. The easy way out of this argument is to state that these
users should be using Tor or something similar. But I still wonder why we
have this double standard of protecting registered users' privacy in
regards to IP addresses and not applying the same for anonymous users, when
simple hashing would do the job.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

signature.asc
Description: Message signed with OpenPGP using AMPGpg
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] User agent policy for bots

2014-07-11 Thread Jeremy Baron

On Jul 11, 2014 9:45 AM, "Marc A. Pelletier"  wrote:
> On 07/11/2014 09:34 AM, John Mark Vandenberg wrote:
> > Could ops confirm they have the username of each logged in edit at
> > their finger tips (i.e. roughly as easy to access as the user-agent)?
> > Pywikibot doesnt permit logged out edits.
>
> We do, after the fact, from the same data Checkusers have access to.

Not if they don't make an edit.

There's lots of options for bots to cause trouble for ops. (including
things that effect all wikis on the cluster, not just the specific one they
were accessing)

> I'm not sure where that talk occured; I have not been made aware of it
> and it didn't filter through the normal ops channels that I've seen.

I believe that's referring to the pywikipedia list.

> I'm a little surprised by Antoine's suggestion that it is important that
> the bot user's information is in the UA string - it doesn't seem useful
> or necessary to me.  Bots shouldn't be editing while logged out in the
> first place, so the bot account will normally always be plain to see.
>
> Obviously, having the user account in the UA would help a bit in
> tracking down errant bots when they happen but that should be a rare
> occurance and we have other methods to use in those cases.

Varnish has access to the cookies, sure. But we log UA string and not
cookies. Or maybe analytics is doing extra logging I didn't notice? If
you're looking at request logs or varnishtop then UA string is a convenient
way (and the standard way we've always suggested to not operators) to
identify the bot.

Imagine if you've identified a specific type of bad request in logs and
they're all from one IP and one UA string. Varnish can easily send an error
for a certain UA string+IP address before it hits the apaches if you need
it to. But if that UA string is generic then you may end up blocking
collateral damage instead of just the one broken bot.

Coren, what you say above is a change from past statements:
I can recall more than a few past conversations about this with mzmcbride,
Tim Starling and others. (Usually comes up when someone comes asking for
whatever app they're writing for a small number of operators. not a big
framework like pwb)

-Jeremy
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] User agent policy for bots

2014-07-11 Thread Marc A. Pelletier

On 07/11/2014 09:34 AM, John Mark Vandenberg wrote:
> Could ops confirm they have the username of each logged in edit at
> their finger tips (i.e. roughly as easy to access as the user-agent)?
> Pywikibot doesnt permit logged out edits.

We do, after the fact, from the same data Checkusers have access to.

> There is some talk that if pywikibot doesnt fix its user-agent string,
> ops may need to block 'the toolserver' - could ops confirm that they
> would usually block a bot account before killing a well known IP range
> like 'the toolserver' (or 'the wmf labs')

That's certainly what *I* would do, and the same applies at least to the
English Wikipedia (where the blocking page clearly points out
"sensitive" ranges which should not be blocked except in cases of dire
emergencies).

I'm not sure where that talk occured; I have not been made aware of it
and it didn't filter through the normal ops channels that I've seen.
I'm a little surprised by Antoine's suggestion that it is important that
the bot user's information is in the UA string - it doesn't seem useful
or necessary to me.  Bots shouldn't be editing while logged out in the
first place, so the bot account will normally always be plain to see.

Obviously, having the user account in the UA would help a bit in
tracking down errant bots when they happen but that should be a rare
occurance and we have other methods to use in those cases.

-- Marc

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] User agent policy for bots

2014-07-11 Thread John Mark Vandenberg

On Fri, Jul 11, 2014 at 6:50 PM, Antoine Musso  wrote:
> Le 11/07/2014 01:09, Amir Ladsgroup a écrit :
>> Hello,
>> As discussions in pywikipedia-l people are not sure whether is necessary to
>> add username of bot operator in user agent or not.
>>
>> In user agent policy 
>> it's mentioned that people need to add contacting information, but it's not
>> clear it's about contacting the tool-maker or tool-user.
>>
>> Can you clarify it?
>
> Hello,
>
> As K. Peachey said, the aim is for Wikimedia operators to be able to
> identify the user running the bot.  The bot framework might be useful.
>
> A suitable user agent could be:
>
>  HasharBot (http://fr.wikipedia.org/wiki/User:Hashar; hashar at free fr)
>
>
> We most probably already have the username in our logs, doesn't harm to
> repeat it in the user-agent.  IRC nickname and email would be nice
> additions and probably save time.

Could ops confirm they have the username of each logged in edit at
their finger tips (i.e. roughly as easy to access as the user-agent)?
Pywikibot doesnt permit logged out edits.

There is some talk that if pywikibot doesnt fix its user-agent string,
ops may need to block 'the toolserver' - could ops confirm that they
would usually block a bot account before killing a well known IP range
like 'the toolserver' (or 'the wmf labs')

IMO it is pretty silly to put the username in the User-Agent for
logged in users who are running adhoc tasks using unmodified pywikibot
code, as they are the user, not its agent. In that scenario, a
distinct version of pywikibot is the agent.  And an email address is
even worse in this scenario.

I do appreciate the need to uniquely identify different user agents,
being any customised code.  Pywikibot already detects which (git)
revision it is running, and includes that in the user agent.  It also
checks versions of files, but I dont think it accurately detects "I am
a customised bot" and definately doesnt include that in the user
agent.  It should.

Also, ideally bots should link to the bot task approval page with
every edit, either in the public edit summary or in the (invisible
except by ops and check-users) user-agent.

Rather than asking bot operators to put an email address in the user
agent, is it OK to have special:emailuser enabled on the bot and
operator?  Or, have a master kill switch on the bot user (task) page?
There is talk of an RFC for 'standardising' how the community can
interact with pywikibot bots, such as disabling bot tasks or the bot
account.
https://gerrit.wikimedia.org/r/#/c/137980/
Checking email is enabled, and ensuring the bot can be easily paused
by 'the community' (inc. ops) strikes me as what is needed, rather
than putting PII into the user *agent*.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Anonymous editors & IP addresses