Re: [Wikitech-l] Request for checking two patches on Gerrit

[Zoran Dori]

Hello everyone,

> With all due respect but i believe it does not scale to use this
> mailing list to ping for individual code reviews.
>
Some peoples have done this in the past, and I wanted to do this also,
because of reasons which I've already provided.

Yes, we really need to fix the issue that getting reviews is too hard and
> too many users seem to filter Gerrit mail but moving that to "everyone
> gets mail about all changes" does not appear to be the solution to me.

Right.

https://phabricator.wikimedia.org/T200987 has some related discussion.
>
 Thanks for providing this.

Best regards,

Zoran Dori
Volunteer on Wikimedia Foundation's projects
E: zorandori4...@gmail.com
*W:* kizule.tk
*Instagram*: https://instagram.com/iamkizule

[image: Banner] 
If the email was sent outside of your working hours, feel free to reply
when you can.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ethical question regarding some code

[Nathan]

 I appreciate that Amir is acknowledging that as neat as this tool sounds,
its use is fraught with risk. The comparison that immediately jumped to my
mind is predictive algorithms used in the criminal justice system to assess
risk of bail jumping or criminal recidividism. These algorithms have been
largely secret, their use hidden, their conclusions non-public. The more we
learn about them, the more deeply flawed it's clear they are. Obviously the
real-world consequences of these tools are more severe in that they
directly lead to the incarceration of many people, but I think the
comparison is illustrative of the risks. It also suggests the type of
ongoing comprehensive review that should be involved in making this tool
available to users.

The potential misuse here to be concerned about is by amateurs with
unsociable intent, or by intended users who are wreckless or ignorant of
the risks. Major governments have the resources to easily build this
themselves, and if they care enough about fingerprinting Wikipedians they
likely already have.

I think if the tool is useful and there's a demand for it, everything about
it - how it works, who uses it, what conclusions and actions are taken as a
result of its use, etc - should be made public. That's the only way we'll
discover the multiple ways in which it will surely eventually be misused.
SPI has been using these 'techniques' in a manual way, or with
unsophisticated tools, for many years. But like any tool, the data fed into
it can be training the system incorrectly. The results it returns can be
misunderstood or intentionally misused. Knowledge of its existence will
lead the most sophisticated to beat it, or intentionally misdirect it.
People who are innocent of any violation of our norms will be harmed by its
use. Please establish the proper cultural and procedural safeguards to
limit the harm as much as possible.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ethical question regarding some code

[QEDK]

I think an important thing to note is that it's public information, so such
a model, either better or worse can easily be built by an AI enthusiast.
The potential for misuse is not much as it's relatively easy to game, and I
don't think that the model's results will hold more water than behaviour
analysis done by a human (which some editors excel at). Theoretically, by
feeding such edits into an assessment system similar to ClueBot and having
expert sockpuppet hunters assess them would result in a much more accurate
and more "dangerous" model, so to say - but with public information, it
shouldn't be closed source and probably only stifles innovation (for e.g.
GPT-3's eventual release).
If the concern is privacy, probably best to dismantle the entire project
but again, someone who wants to can simply put it in the hours required to
do something similar, so not much point and I think by raising this here,
it's probably resulted in the Streisand effect and more people are now
aware of your model and it's possible repercussions, although transparency
is quite integral in all open-source communities. In the end, it all comes
down to your choice, there's no right answer as far as I can tell.

Best,
QEDK

On Fri, Aug 7, 2020, 02:19 John Erling Blad  wrote:

> Nice idea! First time I wrote about this being possible was back in
> 2008-ish.
>
> The problem is quite trivial, you use some observable feature to
> fingerprint an adversary. The adversary can then game the system if the
> observable feature can be somehow changed or modified. To avoid this the
> observable features are usually chosen to be physical properties that can't
> be easily changed.
>
> In this case the features are word and/or relations between words, and then
> the question is “Can the adversary change the choice of words?” Yes he can,
> because the choice of words is not an inherent physical property of the
> user. In fact there are several programs that help users express themselves
> in a more fluent way, and such systems will change the observable features
> i.e. choice of words. The program will move the observable features (the
> words) from one user-specific distribution to another more program-specific
> distribution. You will observe the users a priori to be different, but with
> the program they will be a posteriori more similar.
>
> A real problem is your own poisoning of the training data. That happens
> when you find some subject to be the same as your postulated one, and then
> feed the information back into your training data. If you don't do that
> your training data will start to rot because humans change over time. It is
> bad anyway you do it.
>
> Even more fun is an adversary that knows what you are doing, and tries to
> negate your detection algorithm, or even fool you into believing he is
> someone else. It is after all nothing more than word count and statistics.
> What will you do when someone edits a Wikipedia-page and your system tells
> you “This revision is most likely written by Jimbo”?
>
> Several such programs exist, and I'm a bit perplexed that they are not in
> more use among Wikipedia's editors. Some of them are more aggressive, and
> can propose quite radical rewrites of the text. I use one of them, and it
> is not the best, but still it corrects me all the time.
>
> I believe it would be better to create a system where users are internally
> identified and externally authenticated. (The previous is biometric
> identification, and must adhere to privacy laws.)
>
> On Thu, Aug 6, 2020 at 4:33 AM Amir Sarabadani 
> wrote:
>
> > Hey,
> > I have an ethical question that I couldn't answer yet and have been
> asking
> > around but no definite answer yet so I'm asking it in a larger audience
> in
> > hope of a solution.
> >
> > For almost a year now, I have been developing an NLP-based AI system to
> be
> > able to catch sock puppets (two users pretending to be different but
> > actually the same person). It's based on the way they speak. The way we
> > speak is like a fingerprint and it's unique to us and it's really hard to
> > forge or change on demand (unlike IP/UA), as the result if you apply some
> > basic techniques in AI on Wikipedia discussions (which can be really
> > lengthy, trust me), the datasets and sock puppets shine.
> >
> > Here's an example, I highly recommend looking at these graphs, I compared
> > two pairs of users, one pair that are not sock puppets and the other is a
> > pair of known socks (a user who got banned indefinitely but came back
> > hidden under another username). [1][2] These graphs are based one of
> > several aspects of this AI system.
> >
> > I have talked about this with WMF and other CUs to build and help us
> > understand and catch socks. Especially the ones that have enough
> resources
> > to change their IP/UA regularly (like sock farms, and/or UPEs) and also
> > with the increase of mobile intern providers and the horrible way they
> > assign IP to their users, this can 

Re: [Wikitech-l] Ethical question regarding some code

[John Erling Blad]

For those interested; the best solution as far as I know for this kind of
similarity detection is the Siamese network with RNNs in the first part.
That implies you must extract fingerprints for all likely candidates
(users) and then some to create a baseline. You can not simply claim that
two users (adversary and postulated sock) are the same because they have
edited the same page. It is quite unlikely a user will edit the same page
with a sock puppet, when it is known that such a system is activated.

On Thu, Aug 6, 2020 at 10:49 PM John Erling Blad  wrote:

> Nice idea! First time I wrote about this being possible was back in
> 2008-ish.
>
> The problem is quite trivial, you use some observable feature to
> fingerprint an adversary. The adversary can then game the system if the
> observable feature can be somehow changed or modified. To avoid this the
> observable features are usually chosen to be physical properties that can't
> be easily changed.
>
> In this case the features are word and/or relations between words, and
> then the question is “Can the adversary change the choice of words?” Yes he
> can, because the choice of words is not an inherent physical property of
> the user. In fact there are several programs that help users express
> themselves in a more fluent way, and such systems will change the
> observable features i.e. choice of words. The program will move the
> observable features (the words) from one user-specific distribution to
> another more program-specific distribution. You will observe the users a
> priori to be different, but with the program they will be a posteriori more
> similar.
>
> A real problem is your own poisoning of the training data. That happens
> when you find some subject to be the same as your postulated one, and then
> feed the information back into your training data. If you don't do that
> your training data will start to rot because humans change over time. It is
> bad anyway you do it.
>
> Even more fun is an adversary that knows what you are doing, and tries to
> negate your detection algorithm, or even fool you into believing he is
> someone else. It is after all nothing more than word count and statistics.
> What will you do when someone edits a Wikipedia-page and your system tells
> you “This revision is most likely written by Jimbo”?
>
> Several such programs exist, and I'm a bit perplexed that they are not in
> more use among Wikipedia's editors. Some of them are more aggressive, and
> can propose quite radical rewrites of the text. I use one of them, and it
> is not the best, but still it corrects me all the time.
>
> I believe it would be better to create a system where users are internally
> identified and externally authenticated. (The previous is biometric
> identification, and must adhere to privacy laws.)
>
> On Thu, Aug 6, 2020 at 4:33 AM Amir Sarabadani 
> wrote:
>
>> Hey,
>> I have an ethical question that I couldn't answer yet and have been asking
>> around but no definite answer yet so I'm asking it in a larger audience in
>> hope of a solution.
>>
>> For almost a year now, I have been developing an NLP-based AI system to be
>> able to catch sock puppets (two users pretending to be different but
>> actually the same person). It's based on the way they speak. The way we
>> speak is like a fingerprint and it's unique to us and it's really hard to
>> forge or change on demand (unlike IP/UA), as the result if you apply some
>> basic techniques in AI on Wikipedia discussions (which can be really
>> lengthy, trust me), the datasets and sock puppets shine.
>>
>> Here's an example, I highly recommend looking at these graphs, I compared
>> two pairs of users, one pair that are not sock puppets and the other is a
>> pair of known socks (a user who got banned indefinitely but came back
>> hidden under another username). [1][2] These graphs are based one of
>> several aspects of this AI system.
>>
>> I have talked about this with WMF and other CUs to build and help us
>> understand and catch socks. Especially the ones that have enough resources
>> to change their IP/UA regularly (like sock farms, and/or UPEs) and also
>> with the increase of mobile intern providers and the horrible way they
>> assign IP to their users, this can get really handy in some SPI ("Sock
>> puppet investigation") [3] cases.
>>
>> The problem is that this tool, while being built only on public
>> information, actually has the power to expose legitimate sock puppets.
>> People who live under oppressive governments and edit on sensitive topics.
>> Disclosing such connections between two accounts can cost people their
>> lives.
>>
>> So, this code is not going to be public, period. But we need to have this
>> code in Wikimedia Cloud Services so people like CUs in other wikis be able
>> to use it as a web-based tool instead of me running it for them upon
>> request. But WMCS terms of use explicitly say code should never be
>> closed-source and this is our principle. What should we 

Re: [Wikitech-l] Ethical question regarding some code

[John Erling Blad]

Nice idea! First time I wrote about this being possible was back in
2008-ish.

The problem is quite trivial, you use some observable feature to
fingerprint an adversary. The adversary can then game the system if the
observable feature can be somehow changed or modified. To avoid this the
observable features are usually chosen to be physical properties that can't
be easily changed.

In this case the features are word and/or relations between words, and then
the question is “Can the adversary change the choice of words?” Yes he can,
because the choice of words is not an inherent physical property of the
user. In fact there are several programs that help users express themselves
in a more fluent way, and such systems will change the observable features
i.e. choice of words. The program will move the observable features (the
words) from one user-specific distribution to another more program-specific
distribution. You will observe the users a priori to be different, but with
the program they will be a posteriori more similar.

A real problem is your own poisoning of the training data. That happens
when you find some subject to be the same as your postulated one, and then
feed the information back into your training data. If you don't do that
your training data will start to rot because humans change over time. It is
bad anyway you do it.

Even more fun is an adversary that knows what you are doing, and tries to
negate your detection algorithm, or even fool you into believing he is
someone else. It is after all nothing more than word count and statistics.
What will you do when someone edits a Wikipedia-page and your system tells
you “This revision is most likely written by Jimbo”?

Several such programs exist, and I'm a bit perplexed that they are not in
more use among Wikipedia's editors. Some of them are more aggressive, and
can propose quite radical rewrites of the text. I use one of them, and it
is not the best, but still it corrects me all the time.

I believe it would be better to create a system where users are internally
identified and externally authenticated. (The previous is biometric
identification, and must adhere to privacy laws.)

On Thu, Aug 6, 2020 at 4:33 AM Amir Sarabadani  wrote:

> Hey,
> I have an ethical question that I couldn't answer yet and have been asking
> around but no definite answer yet so I'm asking it in a larger audience in
> hope of a solution.
>
> For almost a year now, I have been developing an NLP-based AI system to be
> able to catch sock puppets (two users pretending to be different but
> actually the same person). It's based on the way they speak. The way we
> speak is like a fingerprint and it's unique to us and it's really hard to
> forge or change on demand (unlike IP/UA), as the result if you apply some
> basic techniques in AI on Wikipedia discussions (which can be really
> lengthy, trust me), the datasets and sock puppets shine.
>
> Here's an example, I highly recommend looking at these graphs, I compared
> two pairs of users, one pair that are not sock puppets and the other is a
> pair of known socks (a user who got banned indefinitely but came back
> hidden under another username). [1][2] These graphs are based one of
> several aspects of this AI system.
>
> I have talked about this with WMF and other CUs to build and help us
> understand and catch socks. Especially the ones that have enough resources
> to change their IP/UA regularly (like sock farms, and/or UPEs) and also
> with the increase of mobile intern providers and the horrible way they
> assign IP to their users, this can get really handy in some SPI ("Sock
> puppet investigation") [3] cases.
>
> The problem is that this tool, while being built only on public
> information, actually has the power to expose legitimate sock puppets.
> People who live under oppressive governments and edit on sensitive topics.
> Disclosing such connections between two accounts can cost people their
> lives.
>
> So, this code is not going to be public, period. But we need to have this
> code in Wikimedia Cloud Services so people like CUs in other wikis be able
> to use it as a web-based tool instead of me running it for them upon
> request. But WMCS terms of use explicitly say code should never be
> closed-source and this is our principle. What should we do? I pay a
> corporate cloud provider for this and put such important code and data
> there? We amend the terms of use to have some exceptions like this one?
>
> The most plausible solution suggested so far (thanks Huji) is to have a
> shell of a code that would be useless without data, and keep the code that
> produces the data (out of dumps) closed (which is fine, running that code
> is not too hard even on enwiki) and update the data myself. This might be
> doable (which I'm around 30% sure, it still might expose too much) but it
> wouldn't cover future cases similar to mine and I think a more long-term
> solution is needed here. Also, it would reduce the bus factor to 

Re: [Wikitech-l] Ethical question regarding some code

[Gergő Tisza]

Technically, you can make the tool open-source and keep the source code
secret. That solves the maintenance problem (others who get access can
legally modify). Of course, you'd have to trust everyone with access to the
files to not publish them which they would be technically entitled to
(unless there is some NDA-like mechanism).

Transparency and auditability wouldn't be fulfilled just by making the code
public, anyway; they need to be solved by tool design (keeping logs,
providing feedback options for the users, trying to expose the components
of the decision as much as possible).

I'd agree with Bawolff though that there is probably no point in going to
great lengths to keep details secret as creating a similar tool is probably
not that hard. You can build some assumptions into the tool which are
nontrivial to fulfill outside Toolforge (e.g. use the replicas instead of
dumps) to make running it require an effort, at least.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ethical question regarding some code

[Thiemo Kreuz]

I'm afraid I have to agree with what AntiCompositeNumber wrote. When
you set up infrastructure to fight abuse – no matter if that
infrastructure is a technical barrier like a captcha, a tool that
"blames" people for being sock puppets, or a law – it will affect
*all* users, not only the abusers. What you need to think about is not
if what you do is right or wrong, but if there is still an acceptable
balance between your intended positive effects, and the unavoidable
negative effects.

That said, I'm very happy to see something like this being discussed
that early. This doesn't always happen. Does anyone still remember
discussing "Deep User Inspector"[1][2] in 2013?

Having read what was already said about "harm", I feel there is
something missing: AI based tools always have the potential to cause
harm simply because people don't really understand what it means to
work with such a tool. For example, when the tool says "there is a 95%
certainty this is a sock puppet", people will use this as "proof",
totally ignoring the fact that the particular case they are looking at
could as well be within the 5%. This is the reason why I believe such
a tool can not be a toy, open for anyone to play around with, but
needs trained users.

TL;DR: Closed source? No. Please avoid at all costs. Closed databases? Sure.

Best
Thiemo

[1] https://ricordisamoa.toolforge.org/dui/
[2] https://meta.wikimedia.org/wiki/User_talk:Ricordisamoa#Deep_user_inspector

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l