Exactly. And the newest shard can also be kept small (e.g. maybe just last 12h is OK to hit first and dig deeper only if you can't find enough stories in the last 12h), which means it will fit in memory and be crazy fast.
Otis -- Solr & ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Wed, Jul 3, 2013 at 10:11 AM, Mark Miller <markrmil...@gmail.com> wrote: > > On Jul 3, 2013, at 7:47 AM, Erick Erickson <erickerick...@gmail.com> wrote: > >> Usually most people >> care about today's news, and a hot story will >> generate lots of queries, all of which are serviced >> by today's shard. > > That's really the whole point though - rather than slamming your whole > cluster with every search, the majority of people are just searching today - > which will have only a fraction of the data and will be able to hold up very > well to a large load. This is also how you can do really fast NRT on a huge > data set - it only has to happen on todays shard. > > News sites have been using the trick forever with Lucene. > > - Mark
