Pardon the length of the question. I have an index with 100 million docs (lucene not solr) and term queries (A*, A AND B* type queries) return pretty quickly (2 -4 secs) and I pick the lucene docIds up pretty quickly with a collector. This is good for us since we take the docIds and do further filtering based on another database we maintain whose record ids match with the stored lucene doc ids and we are able to do what we want. I know that depending on the lucene doc id value is not a good thing, since after delete/merge/optimize, the doc ids may change and if that was to happen, our other datastore will not line up with lucene doc index and chaps will ensue. Thus we do not optimize the index etc.
My question is what is the fastest way I can gather 1 field value from the docs which are found to match the query? Is there any way to do this as fast as (or at least not much slower) I am able to collect the lucene docids? I want to get away from depending on the "lucene docids not changing" if possible. Thanks for any suggestions.
