On Thu, Jun 20, 2013 at 1:08 AM, Nathan Kurz <[email protected]> wrote:
> Mike McCandless just wrote a blog post on speeding up Lucene with some
> C++ code:
> http://blog.mikemccandless.com/2013/06/screaming-fast-lucene-searches-using-c.html
Note that Mike attributes the speedups to three possible factors:
* Algorithmic changes. For example, lucene-c-boost sometimes uses
BooleanScorer where Lucene is using BooleanScorer2. Really we need to
fix Lucene to do similar algorithmic changes (when they are faster).
In particular, all of the OrXX queries that include a Not clause, as
well as IntNRQ in the above results, benefit from algorithmic changes.
* Code specialization: lucene-c-boost implements the searches as big
hardwired scary-looking functions, removing all the nice Lucene
abstractions. While abstractions are obviously necessary in Lucene,
they unfortunately add run-time cost overhead, so removing these
abstractions gives some gains.
* C++ vs java.
It's not at all clear how much of the gains are due to which part; really
I need to create the "matching" specialized Java sources to do a more pure
test.
The part about Lucene's original BooleanScorer being faster than
BooleanScorer2 under many circumstances is something that's been known for a
long time.
* The original BooleanScorer is faster, but cannot support more than 32
sub-scorers and does not support skipping under advance().
* BooleanScorer2 is slower, but can support an arbitrary number of
sub-scorers and supports advance().
We discussed the matter on this dev list a few months ago in the context of
PFOR-delta, and I proposed a Bulk_Score() method inspired by the original
Lucene BooleanScorer's BucketTable, MonetDB, and your "interchange
format".
http://markmail.org/message/tdlhftdngtlxezdv
> I think that Nick is right that the Perl-centric view of Lucy isn't
> helping us. Lucy is designed to be sleek and fast, things Perl
> decidedly is not.
The code that runs while Lucy's Matchers are iterating is completely
disconnected from Perl. Perl is not a factor.
That's a feature of Clownfish.
> What would we need to do to post great numbers on this same Wikipedia
> benchmark?
Well, one necessary step would be to port Lucene's benchmarking suite (it's
big, like everything Lucene) or write or own.
http://s.apache.org/Myn (viewvc for Lucene 4.3.1 benchmark suite)
As far as changing up our own matching code, I think that bulk matching which
avoids method calls on inner loops and exploits the techniques used by MonetDB
has the potential to be a huge leap forward.
There's also another factor we might consider. Clownfish is designed to be
paired with a dynamic language host, but thanks to Nick's efforts, it can now
be used standalone as well. We might want to offer a whole-application
compilation mode which is designed for maxing out performance in a
free-standing compiled application. In the near-to-medium term, that might
mean that virtual method offsets and instance variable offsets are defined as
macro constants rather than global variables. In the long term, it could mean
things like link-time optimization, inlining of simple methods like accessors,
and so on.
> Where do we come out now?
Pretty sure we're behind, just like our FAQ says. Dunno by how much.
> Do others think it's worthwhile to try to improve our reputation in this
> direction?
Well, that's a funny way of putting it. I think it would be very worthwhile
to increase the speed of our Matchers -- but not for the sake of competing
with Lucene. Rather, it's worth doing for the sake of our users, or because
we love the engineering challenges of working on speed.
Marvin Humphrey
