On Wed, Jun 8, 2016 at 4:14 PM Erik Bernhardson <[email protected]> wrote:
> On Wed, Jun 8, 2016 at 3:28 PM, Chad <[email protected]> wrote: > > > On Wed, Jun 8, 2016 at 3:18 PM Deborah Tankersley < > > [email protected]> > > wrote: > > > > > Hi Romaine, > > > > > > Yes, the search box on the wikipedia.org portal only shows results for > > the > > > language that you're currently searching in (which can be changed by > > > selecting a new language in the dropdown located in the search box). We > > > don't yet have the ability to search in all wikis at once but we'd like > > to > > > have that capability in the future. > > > > > > > > Actually we do have the ability to search all wikis at once, it's just > > really > > really slow because of how Nik and I chose to setup the indexes ~4 years > > ago. This could be fixed though :) > > > > > Technically possible, yes. Doable with the current architecture and query > load though, not really feasible. The current architecture (as i'm sure > you're aware) would mean having the elasticsearch servers convert what is > currently a search against 1 index into a search against 900 indexes. Many > of these indexes are small, but a 900x increase in volume is nothing to > scoff at. We do +- 31M full text searches a day, so it's not a tiny amount > of work. It would be asking our servers to process an extra 28 billion > index searches per day. > > We are doing some initial work on figuring out how to reduce this count. > Over the next two quarters discovery will be looking into how to merge > indices of the same language (eswiki, esnews, eswikivoyage, eswiktionary, > eswikiversity, eswikisource, eswikiquote) into a single index per language, > which will make searching between projects in the same language performant, > and will hopefully pave the way for future architecture changes that will > allow more native cross-project searches. Look for this in the coming > months :) > > Awesome! This was always our goal but some early architecture decisions we made has made it harder than it needed to be for the scaling reasons you point out. Great to see work happening towards resolving this and making real cross- wiki searches much closer to reality :) -Chad _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
