tokee commented on issue #754: LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits URL: https://github.com/apache/lucene-solr/pull/754#issuecomment-508818049 I don't know if I have much to add to https://sbdevel.wordpress.com/2015/10/05/speeding-up-core-search/ that Atri linked to on the JIRA: Yes, it's definitely possible to do tricks with large result sets. Especially with the simple "just sort on score"-case where the really large wins in my book is less GC pressure by using a single `long[]` to hold the structure instead of a gazillion small objects. Here's the but: I haven't pursued it further as we (Royal Danish Library) have little use for it. Being able to handle large result sets in a single shards does not help much with multi-shard setups, where the merging node is likely to blow up. An iterative approach, such as `cursorMark` or Solr's `export`, is less prone to surprises and scales indefinitely. That being said, I won't stand in the way of building a fine foot blowing gun - for some use cases it would be a great win.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
