Are you using a Hits object to iterate over the results? If so, you are re-executing the query every 100 docs or so under the covers, and if there are many results, this is very bad.
If this is the case, you want to use a TopDocs or HitCollector to iterate through the entire result set. Of course, this could be entirely irrelevant to your problem...... My experience has been that "fetching the doc" isn't a problem speed-wise, but I have no idea whether your app is different enough from mine to matter in this regard. Best Erick