I came across this post today: http://techcrunch.com/2010/10/06/new-twitter-search
And continued reading these: http://techcrunch.com/2010/10/06/twitter-search-lives/ >From the latter: Modified Lucene Lucene is great, but in its current form it has several shortcomings for real-time search. That’s why we rewrote big parts of the core in-memory data structures, especially the posting lists, while still supporting Lucene’s standard APIs. This allows us to use Lucene’s search layer almost unmodified. Some of the highlights of our changes include: - significantly improved garbage collection performance - lock-free data structures and algorithms - posting lists, that are traversable in reverse order - efficient early query termination We believe that the architecture behind these changes involves several interesting topics that pertain to software engineering in general (not only search). We hope to continue to share more on these improvements. And, before you ask, we’re planning on contributing all these changes back to Lucene; some of which have already made it into Lucene’s trunk and its new realtime branch. And as you can read at the bottom of the last post, Michael B. is behind all this :-). FYI Shai
