Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[WereSpielChequers]

I use search to find typos and misused words, so I'm guilty of some of the
gibberish looking searches
<https://en.wikipedia.org/wiki/User:WereSpielChequers/searches>.

If we are concerned that some common searches could have Privacy
implications, why not create it as a deleted page and announce its
(non)existence on the admins noticeboard?

WSC

On 15 July 2016 at 19:25, <wikimedia-l-requ...@lists.wikimedia.org> wrote:

> Send Wikimedia-l mailing list submissions to
> wikimedia-l@lists.wikimedia.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> or, via email, send a message with subject or body 'help' to
> wikimedia-l-requ...@lists.wikimedia.org
>
> You can reach the person managing the list at
> wikimedia-l-ow...@lists.wikimedia.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Wikimedia-l digest..."
>
>
> Today's Topics:
>
>1. Re: [discovery] Fwd: Improving search (sort of) (Dan Garry)
>2. Re: [discovery] Fwd: Improving search (sort of) (James Heilman)
>3. Re: [discovery] Fwd: Improving search (sort of) (James Heilman)
>4. Re: [discovery] Fwd: Improving search (sort of) (Robert Fernandez)
>5. Re: [discovery] Fwd: Improving search (sort of) (Nathan)
>
>
> --
>
> Message: 1
> Date: Fri, 15 Jul 2016 09:05:54 -0700
> From: Dan Garry <dga...@wikimedia.org>
> To: Wikimedia Mailing List <wikimedia-l@lists.wikimedia.org>
> Cc: A public mailing list about Wikimedia Search and Discovery
>     projects <discov...@lists.wikimedia.org>, Trey Jones
> <tjo...@wikimedia.org>
> Subject: Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)
> Message-ID:
> <
> caow03mhsgoww-gad6udjs_onva8zniuykccrp2evok1b+2d...@mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On 15 July 2016 at 08:44, James Heilman <jmh...@gmail.com> wrote:
> >
> > Thanks for the in depth discussion. So if the terms people are using that
> > result in "zero search results" are typically gibberish why do we care if
> > 30% of our searches result in "zero search results"? A big deal was made
> > about this a while ago.
> >
>
> Good question! I originally used to say that it was my aspiration that
> users should never get zero results when searching Wikipedia. As a result
> of Trey's analysis, I don't say that any more. ;-) There are many
> legitimate cases where users should get zero results. However, there are
> still tons of examples of where giving users zero results is incorrect;
> "jurrasic world" was a prominent example of that.
>
> It's still not quite right to say that *all* the terms that people use to
> get zero results are gibberish. There is an extremely long tail
> <https://en.wikipedia.org/wiki/Long_tail> of zero results queries that
> aren't gibberish, it's just that the top 100 are dominated by gibberish.
> This would mean we'd have to release many, many more than the top 100,
> which significantly increases the risk of releasing personal information.
>
>
> > If one was just to look at those search terms that more than 100 IPs
> > searched for would that not remove the concerns about anonymity? One
> could
> > also limit the length of the searches displaced to 50 characters. And
> just
> > provide the first 100 with an initial human review to make sure we are
> not
> > miss anything.
> >
>
> The problem with this is that there are still no guarantees. What if you
> saw the search query "DF198671E"? You might not think anything of it, but I
> would recognise it as an example of a national insurance number
> <https://en.wikipedia.org/wiki/National_Insurance_number>, the British
> equivalent of a social security number [1]. There's always going to be the
> potential that we accidentally release something sensitive when we release
> arbitrary user input, even if it's manually examined by humans.
>
> So, in summary:
>
>- The top 100 zero results queries are dominated by gibberish.
>- There's a long tail of zero results queries, meaning we'd have to
>reduce many more than the top 100.
>- Manually examining the top zero results queries is not a foolproof way
>of eliminating personal data since it's arbitrary user input.
>
> I'm happy to answer any questions. :-)
>
> Thanks,
> Dan
>
> [1]: Don't panic, this example national insurance number is actually
> invalid. ;-)
>
> --
> Dan Garry
> Lead Product Manager, Di

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[Nathan]

How hard would it be to ask for search feedback on search results, perhaps
piloting with some small subset of zero-result searches? For 1/1000 ZRRs,
prompt the user to provide some type of useful information about why there
should be results, or if there ought to be, or what category of information
the searcher was looking for, etc. You'd get junk and noise, but it might
be one way to filter out a lot of the gibberish. You could also ask people
to agree to make their failed search part of a publicly visible list,
although this could of course be gamed.
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[Robert Fernandez]

> If I can get a
few hundred people to click on a link like this
,
I can get any message I want on that list. (Curious? Did you click?) The
message could be less anonymous and much more obnoxious, obviously

They could vandalize any one of over ten million pages on the English
Wikipedia and get the same result.  We should be conscious of the
dangers but we can easily route around them like we do with other
kinds of vandalism.

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[James Heilman]

Forwarded at the request of Trey Jones

Hey James,

When we first started looking at zero results rate (ZRR), it was an easy
metric to calculate, and it was surprisingly high. We still look at ZRR
 because it is so
easy to measure, and anything that improves it is probably a net positive
(note the big dip when the new completion suggester was deployed!!), but we
have more complex metrics that we prefer. There's user engagement
/augmented
clickthroughs, which combines clicks and dwell time and other user
activity. We also use historical click data in a metric that improves when
we move clicked-on results higher in the results list, which we use with
the Relevance Forge

.

And I didn't mean to give the impression that *most* zero-results queries
are gibberish, though many, many are. And that was something we didn't
really know a year ago. There are also non-gibberish results that correctly
get zero results, like most DOI

and
many media player

queries.
We also see a lot of non-notable (not-yet-notable?) public figures (local
bands, online artists, youtube musicians), and sometimes just random names.

The discussion in response to Dan's original comment in Phab mentions some
approaches to reduce the risk of automatically releasing private info, but
I still take an absolute stand against unreviewed release. If I can get a
few hundred people to click on a link like this
,
I can get any message I want on that list. (Curious? Did you click?) The
message could be less anonymous and much more obnoxious, obviously.

50 character limits won't stop emails and phone numbers from making the
list (which invites spam and cranks). Those can be filtered, but not
perfectly.

I've only looked at these top lists by day in the past, but on that time
scale the top results are usually under 1000 count (and that includes IP
duplicates), so the list of queries with 100 IPs might also be very small.

As I said, I'm happy to do the data slogging to try this in a better
fashion if this task is prioritized, and I'd be happy to be wrong about the
quality of the results, but I'm still not hopeful.

—Trey

Trey Jones
Software Engineer, Discovery
Wikimedia Foundation


On Fri, Jul 15, 2016 at 10:19 AM, James Heilman  wrote:

> The "jurrasic world" example is a good one as it was "fixed" by User:Foxj
> adding a redirect
> https://en.wikipedia.org/w/index.php?title=Jurrasic_world=history
>
> Agree we would need to be careful. The chance of many different IPs all
> searching for "DF198671E" is low but I agree not zero and we would need
> to have people run the results before they are displayed.
>
> I guess the question is how much work would it take to look at this sort
> of data for more examples like "jurrasic world"?
>
> James
>
> On Fri, Jul 15, 2016 at 10:05 AM, Dan Garry  wrote:
>
>> On 15 July 2016 at 08:44, James Heilman  wrote:
>> >
>> > Thanks for the in depth discussion. So if the terms people are using
>> that
>> > result in "zero search results" are typically gibberish why do we care
>> if
>> > 30% of our searches result in "zero search results"? A big deal was made
>> > about this a while ago.
>> >
>>
>> Good question! I originally used to say that it was my aspiration that
>> users should never get zero results when searching Wikipedia. As a result
>> of Trey's analysis, I don't say that any more. ;-) There are many
>> legitimate cases where users should get zero results. However, there are
>> still tons of examples of where giving users zero results is incorrect;
>> "jurrasic world" was a prominent example of that.
>>
>> It's still not quite right to say that *all* the terms that people use to
>> get zero results are gibberish. There is an extremely long tail
>>  of zero results queries that
>> aren't gibberish, it's just that the top 100 are dominated by gibberish.
>> This would mean we'd have to release many, many more than the top 100,
>> which significantly increases the risk of releasing personal information.
>>
>>
>> > If one was just to look at those search terms that more than 100 IPs
>> > searched for would that not remove the concerns about anonymity? One
>> could
>> > also limit the length of the searches displaced to 50 characters. And
>> just
>> > provide the first 100 with an initial human review to make sure we are
>> not
>> > miss anything.
>> >
>>
>> The problem with this is that there are 

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[James Heilman]

The "jurrasic world" example is a good one as it was "fixed" by User:Foxj
adding a redirect
https://en.wikipedia.org/w/index.php?title=Jurrasic_world=history

Agree we would need to be careful. The chance of many different IPs all
searching for "DF198671E" is low but I agree not zero and we would need to
have people run the results before they are displayed.

I guess the question is how much work would it take to look at this sort of
data for more examples like "jurrasic world"?

James

On Fri, Jul 15, 2016 at 10:05 AM, Dan Garry  wrote:

> On 15 July 2016 at 08:44, James Heilman  wrote:
> >
> > Thanks for the in depth discussion. So if the terms people are using that
> > result in "zero search results" are typically gibberish why do we care if
> > 30% of our searches result in "zero search results"? A big deal was made
> > about this a while ago.
> >
>
> Good question! I originally used to say that it was my aspiration that
> users should never get zero results when searching Wikipedia. As a result
> of Trey's analysis, I don't say that any more. ;-) There are many
> legitimate cases where users should get zero results. However, there are
> still tons of examples of where giving users zero results is incorrect;
> "jurrasic world" was a prominent example of that.
>
> It's still not quite right to say that *all* the terms that people use to
> get zero results are gibberish. There is an extremely long tail
>  of zero results queries that
> aren't gibberish, it's just that the top 100 are dominated by gibberish.
> This would mean we'd have to release many, many more than the top 100,
> which significantly increases the risk of releasing personal information.
>
>
> > If one was just to look at those search terms that more than 100 IPs
> > searched for would that not remove the concerns about anonymity? One
> could
> > also limit the length of the searches displaced to 50 characters. And
> just
> > provide the first 100 with an initial human review to make sure we are
> not
> > miss anything.
> >
>
> The problem with this is that there are still no guarantees. What if you
> saw the search query "DF198671E"? You might not think anything of it, but I
> would recognise it as an example of a national insurance number
> , the British
> equivalent of a social security number [1]. There's always going to be the
> potential that we accidentally release something sensitive when we release
> arbitrary user input, even if it's manually examined by humans.
>
> So, in summary:
>
>- The top 100 zero results queries are dominated by gibberish.
>- There's a long tail of zero results queries, meaning we'd have to
>reduce many more than the top 100.
>- Manually examining the top zero results queries is not a foolproof way
>of eliminating personal data since it's arbitrary user input.
>
> I'm happy to answer any questions. :-)
>
> Thanks,
> Dan
>
> [1]: Don't panic, this example national insurance number is actually
> invalid. ;-)
>
> --
> Dan Garry
> Lead Product Manager, Discovery
> Wikimedia Foundation
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 




-- 
James Heilman
MD, CCFP-EM, Wikipedian

The Wikipedia Open Textbook of Medicine
www.opentextbookofmedicine.com
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[Dan Garry]

On 15 July 2016 at 08:44, James Heilman  wrote:
>
> Thanks for the in depth discussion. So if the terms people are using that
> result in "zero search results" are typically gibberish why do we care if
> 30% of our searches result in "zero search results"? A big deal was made
> about this a while ago.
>

Good question! I originally used to say that it was my aspiration that
users should never get zero results when searching Wikipedia. As a result
of Trey's analysis, I don't say that any more. ;-) There are many
legitimate cases where users should get zero results. However, there are
still tons of examples of where giving users zero results is incorrect;
"jurrasic world" was a prominent example of that.

It's still not quite right to say that *all* the terms that people use to
get zero results are gibberish. There is an extremely long tail
 of zero results queries that
aren't gibberish, it's just that the top 100 are dominated by gibberish.
This would mean we'd have to release many, many more than the top 100,
which significantly increases the risk of releasing personal information.


> If one was just to look at those search terms that more than 100 IPs
> searched for would that not remove the concerns about anonymity? One could
> also limit the length of the searches displaced to 50 characters. And just
> provide the first 100 with an initial human review to make sure we are not
> miss anything.
>

The problem with this is that there are still no guarantees. What if you
saw the search query "DF198671E"? You might not think anything of it, but I
would recognise it as an example of a national insurance number
, the British
equivalent of a social security number [1]. There's always going to be the
potential that we accidentally release something sensitive when we release
arbitrary user input, even if it's manually examined by humans.

So, in summary:

   - The top 100 zero results queries are dominated by gibberish.
   - There's a long tail of zero results queries, meaning we'd have to
   reduce many more than the top 100.
   - Manually examining the top zero results queries is not a foolproof way
   of eliminating personal data since it's arbitrary user input.

I'm happy to answer any questions. :-)

Thanks,
Dan

[1]: Don't panic, this example national insurance number is actually
invalid. ;-)

-- 
Dan Garry
Lead Product Manager, Discovery
Wikimedia Foundation
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] [discovery] Fwd: Improving search (sort of)

[James Heilman]

Hey Trey

Thanks for the in depth discussion. So if the terms people are using that
result in "zero search results" are typically gibberish why do we care if
30% of our searches result in "zero search results"? A big deal was made
about this a while ago.

If one was just to look at those search terms that more than 100 IPs
searched for would that not remove the concerns about anonymity? One could
also limit the length of the searches displaced to 50 characters. And just
provide the first 100 with an initial human review to make sure we are not
miss anything.

James

On Fri, Jul 15, 2016 at 9:31 AM, Trey Jones  wrote:

> Pine, thanks for the forward. Regulars on the Discovery list may know me,
> but James probably does not. I've manually reviewed tens of thousands of
> generally poorly performing queries (fewer than 3 results) and skimmed
> hundreds of thousands more from many of the top 20 Wikipedias—and to a
> lesser extent other projects—over the year I've been at the WMF and in
> Discovery. You can see my list of write ups here
> .
>
> So I want to say that this is an awesome idea—which is why many people
> have thought of it. It was apparently one of the first ideas the Discovery
> department had when they formed (see Dan's notes linked below). It was also
> one of the first ideas I had when I joined Discovery a few months later.
>
> Dan Garry's notes on T8373
>  and the following
> discussion pretty much quash the idea of automated extraction and
> publication from a privacy perspective. People not only divulge their own
> personal information, they also divulge other people's personal
> information. One example: some guy outside the U.S. was methodically
> searching long lists of real addresses in Las Vegas. I will second Dan's
> comments in the T8373 discussion; all kinds of personal data end up in
> search queries. A dump of search queries *was* provided in September 2012
> ,
> but had to be withdrawn over privacy concerns.
>
> Another concern for auto-published data: never underestimate the power of
> random groups of bored people on the internet. 4chan decided to arrange
> Time Magazine poll results
> 
>  so
> the first letter spelled out a weird message. It would be easy for 4chan,
> Reddit, and other communities to get any message they want on that list if
> they happened to notice that it existed. See also Boaty McBoatface
>  and Mountain
> Dew "Diabeetus"
> 
> (which is not at all the worst thing on *that* list). We don't want to
> have to try to defend against that.
>
> In my experience, the quality of what's actually there isn't that great.
> One of my first tasks when I joined Discovery was to look at daily lists of
> top 100 zero-results queries that had been gathered automatically. I was
> excited by this same idea. The top 100 zero-results query list was a
> wasteland. (Minimal notes on some of what I found are here
> .)
> We could make it better by focusing on human-ish searchers, using basic
> bot-exclusion techniques
> ,
> ignoring duplicates from the same IP, and such, but I don't think it would
> help. And while Wikipedia is not for children, there could be an annoying
> amount of explicit adult material on the list, too. We would probably find
> some interesting spellings of Facebook and WhatsApp, though.
>
> If we're really excited about this, I could imagine using better
> techniques to pull zero-results queries and see if anything good is in
> there, but we'd have to commit to some sort of review before we publish it.
> For example, Discernatron  data, after
> consulting with legal, is reviewed independently by two people, who then
> have to reconcile any discrepancies, before being made public. So I think
> we'd need an ongoing commitment to have at least two people under NDA who
> would review any list before publication. 500-600 queries takes a couple
> hours per person (we’ve done that for the Discernatron), so the top 100
> would probably be less than an hour. I'd even be willing to help with the
> review (as I am for Discernatron) if we found there was something useful in
> there—but I'm not terribly hopeful. We'd also need more people to
> efficiently and effectively review queries for other languages if we wanted
> to extend this