Joseph Obernberger [j...@lovehorsepower.com] wrote: [HDFS, 9M docs, 2.9TB, 22 shards, 11 bare metal boxes]
> A typical query takes about 7 seconds to run, but we also do faceting > and clustering. Those can take in the 3 - 5 minute range depends on > what was queried, but can be as little as 10 seconds. The index contains > about 100 fields. 7 seconds without faceting seems like a long time. I am guessing your 3M daily updates are spread throughout the day, instead of being a nightly batch job? How many concurrent searches are you handling? We have no experience with HDFS for Solr indexes, but a quick check indicates that it is not a good fit for Solr. At least not out of the box: http://hbase.apache.org/book.html#perf.hdfs.curr We did at one point try to use networked storage for our index. That meant 1/3 performance, compared to local storage, but of course your mileage will vary. As you are looking into ways of improving performance, what about testing the performance difference with local storage (SSD of course)? - Toke Eskildsen
