[Wikimedia-search] Enable or disable full text search query rewriting by default for API clients.

[Erik Bernhardson]

We have a new feature for web requests that rewrites zero result queries
into a new search that might have results.  I've started porting this same
feature over to API clients so it has a larger effect on our zero results
rate, but code review has turned up some indecision on if this should be
enabled or disabled by default in the API.  Either way the feature will be
toggleable.

I thought we should open this up to a larger audience, are there any
opinions?

Erik B.
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Enable or disable full text search query rewriting by default for API clients.

[Erik Bernhardson]

On Thu, Jul 30, 2015 at 3:29 PM, Adam Baso  wrote:

> Probably a good idea. Is it opt-in or opt-our for the API consumer?
>
> -Adam
>
>
Thats the main question :)  To copy the dialog from gerrit:

> *Anomie wrote:*
> Should the ability to set this flag be exposed to the API somehow?
>
> Or, to avoid changing things for clients, perhaps that should be "should
the
> ability to not set this flag be exposed to the API somehow?" since the
API
> hasn't done any query rewriting before?

> *Ebernhardson wrote:*
> As for the default, The current scope of rewriting is very small, we are
only
> applying it to situations where the original query returned no results
and the
> original query didn't contain any special syntax(such as quotes,
incategory: etc).
> We might investigate doing heavier rewriting in the future (hasn't been
> considered yet), but for this quarter we are focusing specifically on
returning
> answers to queries that return no results.
>
> As this only effects queries that didn't have a result anyways, i think
it should
> be safe to apply as a default and allow api consumers to opt out.

> *Anomie wrote:*
> OTOH, someone might be running a bot that searches for a common typo and
> fixes it. Once all the instances of the typo are fixed, your rewriting
might cause
> it to return search results for the un-typoed version, causing the bot to
at best
> waste bandwidth by fetching many pages that don't need any edit (and in
worse
> cases it might make annoying minor edits or otherwise misbehave).

I thought it might be useful to get more opinions than our two.

Erik B.
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Enable or disable full text search query rewriting by default for API clients.

[Erik Bernhardson]

On Thu, Jul 30, 2015 at 4:51 PM, Legoktm 
wrote:

> On 07/30/2015 02:06 PM, Erik Bernhardson wrote:
> > results rate, but code review has turned up some indecision on if this
> > should be enabled or disabled by default in the API.  Either way the
> > feature will be toggleable.
>
> What gerrit change is this about? Can you provide a link? :)
>
>
The core change is https://gerrit.wikimedia.org/r/#/c/227501/
There is a related cirrus change at
https://gerrit.wikimedia.org/r/#/c/227578/
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

[Wikimedia-search] Completion suggestion API demo

[Erik Bernhardson]

We have been working on a replacement autocompletion API that is more
forgiving than a strict prefix search.  The scoring algorithm's have a long
way to go but we have the first run through of building the completion
index for enwiki so i thought i would share:

Here are a couple examples, feel free to change the text= around to other
things.

http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=white%20house
http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=hotel%20cal
http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=wikim


Don't stress out too much over the scoring yet, we know it needs some work
and have plans to integrate page view information in here to help more
popular articles rise the the top.

Erik B
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Completion suggestion API demo

[Erik Bernhardson]

I ran some zero result rate tests against this API today, it is a huge
reduction in the zero result rate over the existing prefix search.  from
32% to 19% (on a 1% sample of prefix searches for an entire day)

On Wed, Aug 26, 2015 at 12:34 PM, Stas Malyshev 
wrote:

> Hi!
>
> > I uploaded a small HTML page to compare both approaches:
> > http://cirrus-browser-bot.wmflabs.org/suggest.html
>
> This is very cool! From my very short testing, seems that it works
> pretty nicely.
>
> --
> Stas Malyshev
> smalys...@wikimedia.org
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Measuring user user satisfaction while reducing it at the same time?

[Erik Bernhardson]

On Wed, Aug 26, 2015 at 3:58 PM, Chad Horohoe 
wrote:

> Aren't we uncached here anyway? Special pages and all.
>
> -Chad
>
Actually the events we are recording here measure the users interaction
with the pages they found. The current idea is to add a query parameter to
all search results (only for users in the test, from javascript) so that
when they land on that page we know they came from search and start
collecting interaction events.
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Measuring user user satisfaction while reducing it at the same time?

[Erik Bernhardson]

Thats an excellent idea, thanks Adam.  Talked about it w/ max and updated
the patch, i think this will work out just fine.

Erik B.

On Wed, Aug 26, 2015 at 4:23 PM, Adam Baso  wrote:

> It may not be the *appropriate* parameter name to use, but this sort of
> technique may help:
>
> https://wikitech.wikimedia.org/wiki/Provenance
>
> You'd want to look at the *current* VCL in templates/varnish in the
> operations repo to see how it's presently done for *wprov*.
>
> -Adam
>
>
>
> On Wed, Aug 26, 2015 at 4:00 PM, Erik Bernhardson <
> ebernhard...@wikimedia.org> wrote:
>
>> On Wed, Aug 26, 2015 at 3:58 PM, Chad Horohoe 
>> wrote:
>>
>>> Aren't we uncached here anyway? Special pages and all.
>>>
>>> -Chad
>>>
>> Actually the events we are recording here measure the users interaction
>> with the pages they found. The current idea is to add a query parameter to
>> all search results (only for users in the test, from javascript) so that
>> when they land on that page we know they came from search and start
>> collecting interaction events.
>>
>> ___
>> Wikimedia-search mailing list
>> Wikimedia-search@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Measuring user user satisfaction while reducing it at the same time?

[Erik Bernhardson]

(cross-posting to ops@ as requested. This is in regards to an EventLogging
schema[1] applied in javascript to track user behavior on pages they find
via internal search to measure the quality of the search results[2][3]. )

The referrer would be nice to use, but we are trying to track more than
just a single click away from Special:Search. We are trying to track all
pages the user visits (for 10 minutes) by clicking from search results to
one page and then another. The other difficulty here is we are trying to
track only the pages the user clicked to coming from our search, and not a
page they found by repeating their search on google and clicking through to
Wikipedia.

Using the wprov[4] query parameter to declare the provenance as desktop
search should avoid all the issues mentioned above. Varnish strips wprov so
the varnish cache remains unfragmented. The javascript can pickup on the
parameter and log the necessary events.

[1] https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2
[2] https://phabricator.wikimedia.org/T109482
[3] https://gerrit.wikimedia.org/r/#/c/232896/
[4] https://wikitech.wikimedia.org/wiki/Provenance

On Thu, Aug 27, 2015 at 3:10 AM, Giuseppe Lavagetto <
glavage...@wikimedia.org> wrote:

> Hi all,
>
> in determining where a user is coming from, the "referer" header would
> be much more advisable in general, but maybe I'm losing the context
> here.
>
> Also, I think this thread should be x-posted to ops@ and/or relevant
> tickets should be brought to our attention :)
>
> Cheers
>
> Joe
>
> On Thu, Aug 27, 2015 at 9:13 AM, Federico Leva (Nemo)
>  wrote:
> > Max Semenik, 26/08/2015 22:30:
> >>
> >> It does that by appending fromsearch=1 to links for 0.5% of users.
> >
> >
> > See also https://phabricator.wikimedia.org/T106594
> >
> >
> > ___
> > Wikimedia-search mailing list
> > Wikimedia-search@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Completion suggestion API demo

[Erik Bernhardson]

We had to shuffle a few things around in labs, the demo is now running at:

http://suggesty.wmflabs.org/suggest.html

On Thu, Aug 27, 2015 at 11:00 AM, Dan Garry  wrote:

> Our initial tests of this suggestions API are incredibly promising at
> reducing the zero results rate: https://phabricator.wikimedia.org/T109729
>
> More rigorous testing must be done before we can consider replacing
> prefixsearch with the suggestion API.
>
> Thanks!
>
> Dan
>
> On 25 August 2015 at 15:38, Erik Bernhardson 
> wrote:
>
>> We have been working on a replacement autocompletion API that is more
>> forgiving than a strict prefix search.  The scoring algorithm's have a long
>> way to go but we have the first run through of building the completion
>> index for enwiki so i thought i would share:
>>
>> Here are a couple examples, feel free to change the text= around to other
>> things.
>>
>>
>> http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=white%20house
>>
>> http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=hotel%20cal
>>
>> http://cirrus-browser-bot.wmflabs.org/w/api.php?action=cirrus-suggest&limit=100&text=wikim
>>
>>
>> Don't stress out too much over the scoring yet, we know it needs some
>> work and have plans to integrate page view information in here to help more
>> popular articles rise the the top.
>>
>> Erik B
>>
>> ___
>> Wikimedia-search mailing list
>> Wikimedia-search@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>
>
> --
> Dan Garry
> Lead Product Manager, Discovery
> Wikimedia Foundation
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

[Wikimedia-search] Asynchronously calling elasticsearch

[Erik Bernhardson]

The php engine used in prod by the wmf, hhvm, has built in support for
shared (non-preemptive) concurrency via async/await keywords[1][2]. Over
the weekend i spent some time converting the Elastica client library we use
to work asynchronously, which would essentially let us continue on
performing other calculations in the web request while network requests are
processing. I've only ported over the client library[3], not the
CirrusSearch code. Also this is not a complete port, there are a couple
code paths that work but most of the test suite still fails.

The most obvious place we could see a benefit from this is when multiple
queries are issued to elasticsearch from a single web request. If the
second query doesn't depend on the results of the first it can be issued in
parallel. This is actually somewhat common use case, for example doing a
full text and a title search in the same request. I'm wary of making much
of a guess in terms of actual latency reduction we could expect, but maybe
on the order of 50 to 100 ms in cases which we currently perform requests
serially and we have enough work to process. Really its hard to say at this
point.

In addition to making some existing code faster, having the ability to do
multiple network operations in an async manner opens up other possibilities
when we are implementing things in the future.  In closing, this currently
isn't going anywhere it was just something interesting to toy with.  I
think it could be quite interesting to investigate further.

[1] http://docs.hhvm.com/manual/en/hack.async.php
[2] https://phabricator.wikimedia.org/T99755
[2] https://github.com/ebernhardson/Elastica/tree/async
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Asynchronously calling elasticsearch

[Erik Bernhardson]

This would allow them to be run in parallel, yes. Being in separate
databases means extra last-minute checks for existence or security (CYA for
if deletes get missed) are skipped as interwiki links are generated, but
overall not a big deal and an expected loss as part of the interwiki search.

On Wed, Sep 9, 2015 at 10:11 AM, Kevin Smith  wrote:

> Would this help if we wanted to simultaneously search multiple wikis, or
> are those in separate databases so it would have no effect?
>
>
> Kevin Smith
> Agile Coach, Wikimedia Foundation
>
>
> On Wed, Sep 9, 2015 at 5:18 AM, David Causse 
> wrote:
>
>> Thanks Erik!
>>
>> This is very promising and it opens a lot of new possibilities.
>> Guessing the gain is pretty hard but I think we run many small requests
>> where network overhead is quite high compared to the actual work done by
>> elastic. This would definitely help.
>>
>> Le 08/09/2015 21:01, Erik Bernhardson a écrit :
>>
>> The php engine used in prod by the wmf, hhvm, has built in support for
>> shared (non-preemptive) concurrency via async/await keywords[1][2]. Over
>> the weekend i spent some time converting the Elastica client library we use
>> to work asynchronously, which would essentially let us continue on
>> performing other calculations in the web request while network requests are
>> processing. I've only ported over the client library[3], not the
>> CirrusSearch code. Also this is not a complete port, there are a couple
>> code paths that work but most of the test suite still fails.
>>
>> The most obvious place we could see a benefit from this is when multiple
>> queries are issued to elasticsearch from a single web request. If the
>> second query doesn't depend on the results of the first it can be issued in
>> parallel. This is actually somewhat common use case, for example doing a
>> full text and a title search in the same request. I'm wary of making much
>> of a guess in terms of actual latency reduction we could expect, but maybe
>> on the order of 50 to 100 ms in cases which we currently perform requests
>> serially and we have enough work to process. Really its hard to say at this
>> point.
>>
>> In addition to making some existing code faster, having the ability to do
>> multiple network operations in an async manner opens up other possibilities
>> when we are implementing things in the future.  In closing, this currently
>> isn't going anywhere it was just something interesting to toy with.  I
>> think it could be quite interesting to investigate further.
>>
>> [1] http://docs.hhvm.com/manual/en/hack.async.php
>> [2] https://phabricator.wikimedia.org/T99755
>> [2] https://github.com/ebernhardson/Elastica/tree/async
>>
>>
>>
>> ___
>> Wikimedia-search mailing 
>> listWikimedia-search@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>>
>> ___
>> Wikimedia-search mailing list
>> Wikimedia-search@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Smoothing in dashboard(s)

[Erik Bernhardson]

This is awesome. The weekly median for zero results rate change is much
easier to comprehend. Thanks for doing this.

On Thu, Sep 10, 2015 at 9:15 AM, Trey Jones  wrote:

> Smth! It looks great!
>
> Quick question—what's the period of the moving average? Is it a week, or
> more? (A week makes sense, but it looks like more, based on how much data
> drops out at the beginning.) If it's not a week, can I suggest a week? It
> may further smooth out some of the daily bumps to only have one Monday, for
> example, in any given average, since the daily pattern is so extreme.
>
>
> On Thu, Sep 10, 2015 at 12:00 PM, Mikhail Popov 
> wrote:
>
>> Hi all,
>>
>> You can now apply various smoothers to the data on the Search Metrics
>> dashboard (e.g. http://searchdata.wmflabs.org/metrics/#failure_rate).
>> Smoothing can be applied globally or on a per-plot basis.
>>
>> Hopefully yinz will find this new feature useful!
>>
>> Cheers~
>>
>> --
>> *Mikhail Popov* // Data Analyst, Discovery
>> 
>> https://wikimediafoundation.org/
>>
>> *Imagine a world in which every single human being can freely share in
>> the **sum of all knowledge. That's our commitment.* Donate
>> .
>>
>> ___
>> Wikimedia-search mailing list
>> Wikimedia-search@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>
> ___
> Wikimedia-search mailing list
> Wikimedia-search@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

[Wikimedia-search] Page rank

[Erik Bernhardson]

Late last week while looking over our existing scoring methods i was
thinking that while counting incoming links is nice, a couple guys
dominated search with (among other things) a better way to judge the
quality of incoming links, aka PageRank.

PageRank takes a very simple input, it just needs a list of all links
between pages. We happen to already store all of these in elasticsearch. I
wrote a few scripts to suck out the full enwiki graph (~400M edges), ship
it over to stat1002, throw it into hadoop, and crunch it with a few hundred
cores. The end result is a score for every NS_MAIN page in enwiki based on
the quality of incoming links.

I've taken these calculated pagerank's and used them as the scoring method
for search-as-you-type for http://en-suggesty.wmflabs.org.

Overall this seems promising as another scoring metric to integrate to our
search results. Not sure yet how to figure out things like how much weight
does pagerank have in the score? This might be yet another thing where
building out our relevance lab would enable us to make more informed
decisions.

Overall i think some sort of pipeline from hadoop into our scoring system
could be quite useful.  The initial idea seems to be to crunch data in
hadoop, stuff it into a read-only api, and then query it back out at
indexing time in elasticsearch to be held within the ES docs. I'm not sure
what the best way will be, but having a simple and repeatable way to
calculate scoring info in hadoop and ship that into ES will probably become
more and more important.
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Asynchronously calling elasticsearch

[Erik Bernhardson]

Just to follow up here, i've updated the `async` branch of my Elastica
fork, it now completely passes the test suite so might be ready for further
CirrusSearch testing.

On Wed, Sep 9, 2015 at 12:23 PM, Erik Bernhardson <
ebernhard...@wikimedia.org> wrote:

> This would allow them to be run in parallel, yes. Being in separate
> databases means extra last-minute checks for existence or security (CYA for
> if deletes get missed) are skipped as interwiki links are generated, but
> overall not a big deal and an expected loss as part of the interwiki search.
>
> On Wed, Sep 9, 2015 at 10:11 AM, Kevin Smith  wrote:
>
>> Would this help if we wanted to simultaneously search multiple wikis, or
>> are those in separate databases so it would have no effect?
>>
>>
>> Kevin Smith
>> Agile Coach, Wikimedia Foundation
>>
>>
>> On Wed, Sep 9, 2015 at 5:18 AM, David Causse 
>> wrote:
>>
>>> Thanks Erik!
>>>
>>> This is very promising and it opens a lot of new possibilities.
>>> Guessing the gain is pretty hard but I think we run many small requests
>>> where network overhead is quite high compared to the actual work done by
>>> elastic. This would definitely help.
>>>
>>> Le 08/09/2015 21:01, Erik Bernhardson a écrit :
>>>
>>> The php engine used in prod by the wmf, hhvm, has built in support for
>>> shared (non-preemptive) concurrency via async/await keywords[1][2]. Over
>>> the weekend i spent some time converting the Elastica client library we use
>>> to work asynchronously, which would essentially let us continue on
>>> performing other calculations in the web request while network requests are
>>> processing. I've only ported over the client library[3], not the
>>> CirrusSearch code. Also this is not a complete port, there are a couple
>>> code paths that work but most of the test suite still fails.
>>>
>>> The most obvious place we could see a benefit from this is when multiple
>>> queries are issued to elasticsearch from a single web request. If the
>>> second query doesn't depend on the results of the first it can be issued in
>>> parallel. This is actually somewhat common use case, for example doing a
>>> full text and a title search in the same request. I'm wary of making much
>>> of a guess in terms of actual latency reduction we could expect, but maybe
>>> on the order of 50 to 100 ms in cases which we currently perform requests
>>> serially and we have enough work to process. Really its hard to say at this
>>> point.
>>>
>>> In addition to making some existing code faster, having the ability to
>>> do multiple network operations in an async manner opens up other
>>> possibilities when we are implementing things in the future.  In closing,
>>> this currently isn't going anywhere it was just something interesting to
>>> toy with.  I think it could be quite interesting to investigate further.
>>>
>>> [1] http://docs.hhvm.com/manual/en/hack.async.php
>>> [2] https://phabricator.wikimedia.org/T99755
>>> [2] https://github.com/ebernhardson/Elastica/tree/async
>>>
>>>
>>>
>>> ___
>>> Wikimedia-search mailing 
>>> listWikimedia-search@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>>
>>>
>>>
>>> ___
>>> Wikimedia-search mailing list
>>> Wikimedia-search@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>>
>>>
>>
>> ___
>> Wikimedia-search mailing list
>> Wikimedia-search@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
>>
>>
>
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search

Re: [Wikimedia-search] Asynchronously calling elasticsearch

[Erik Bernhardson]

On Mon, Sep 21, 2015 at 8:51 AM, Trey Jones  wrote:

> That's very cool! Have you stress-tested it at all? Like, what happens if
> you search 10 wikipedias at once? (Because you know I want to search 10
> wikis at once. )
>

No issues with a dozen requests at once in the application layer. Our
cluster on the other hand may not be as happy :)  We will have to figure
out how to manage the load this feature could generate.

On Mon, Sep 21, 2015 at 12:56 PM, Stas Malyshev 
wrote:
>
> Cool! I wonder if there's any interaction with the pool counters search
> is using. I.e. would it block on pool counter, use one counter for all
> async work, etc.


Good question. Pool counter is very picky about how you can nest locks. It
actually rejected any attempt to nest locks until nik changed it for the
purposes of allowing us to nest the regular lock inside of a per-user lock.
I'm not sure how we will want to move forward here, something to think
about.
___
Wikimedia-search mailing list
Wikimedia-search@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search