Automaton performance improvements based on Lucene code base
------------------------------------------------------------
Key: NUTCH-1068
URL: https://issues.apache.org/jira/browse/NUTCH-1068
Project: Nutch
Issue Type: Improvement
Reporter: Kirby Bohling
The Lucene team maintains a modified Automaton library cut down to precisely
what they need. It can have significant performance enhancements.
I am attempting to backport and shepherd a patch for the original Automaton
library.
The original Lucene code is here:
http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/util/automaton/
The Lucene code is likely slightly faster, as it includes several micro
optimizations I removed to avoid having to request re-license permission. I
would definitely performance test using the Lucene RegEx vs. the patched code.
The Lucene code also uses code points not characters, which might make a
difference for UTF-16 vs. UTF-32 in obscure cases (I believe the Lucene code
builds a UTF-32 clean DFA for accuracy, and then translates it to a UTF-8 DFA
for performance but I'm not 100% sure. I don't need/use any of that code, and
currently really only worried about ASCII DFAs).
When making heavy use of the NFA-to-DFA transformation, I see a 4x speed up.
It likely has a 1.5-2x speed up for regular expression execution from what I
can tell. The Nutch backend uses this code in a couple of places, and it
likely would lead to performance benefits for those areas.
I will attach my backported version for the Automaton 1.11-7 release. While I
don't own any of the copyright, all of the code is copyrighted under the BSD
license, or the ASF 2.0 license. It is pretty obviously approved for ASF
usage. I am not checking that the patch is usable as I'm not the copyright
holder. If that is an issue, I'll say "yes", I just don't believe I have any
legal standing to do so. I don't want to create licensing issues for the ASF.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
