Over in Solr, there's an open jira regarding some "drift" that has
happened over time between some of the lang specific stopword files that
Solr shipts in it's default configset and the equivilent files that are
provided in the lucene jars (and loaded by the corrisponding lucene
Analyzers via getResourceAsStream()).
That got me thinking about adding some tooling to Solr's build to update
these files automatically when we upgrade Lucene, which got me looking at
what that would invovle, which lead me to realize there seems to be some
inconsistencies in what static default CharArraySets are/aren't "public"
in Lucene Analyzer classes.
For example:
- Most (all?) Analyzer classes that have a default list
of stopwords seem to include a "public static CharArraySet
getDefaultStopSet()"
...but...
- Of the Analyzers that use ElisionFilter, only FrenchAnalyzer has a
"public static final CharArraySet DEFAULT_ARTICLES" -- IrishAnalyzer,
ItalianAnalyzer, & CatalanAnalyzer keep it private
-IrishAnalyzer also has a "private static final CharArraySet HYPHENATIONS"
that's documented as being important to use as stopwrds when using
StandardTokenizer
- DutchAnalyzer has a (private) 'static final CharArrayMap<String>
DEFAULT_STEM_DICT' that it uses with StemmerOverrideFilter
...for my purposes, this is inconvinient, but not insurmountable, but
practically speaking, the bigger concern I have for asking if folks think
these kinds of static "defaults" should always be "public" is because it
seems like any (lucene) user who starts out using something like "new
IrishAnalyzer()" and then decides later that they want to write their own
custom Analyzer to tweak beahvior would probably prefer to do this...
protected TokenStreamComponents createComponents(String fieldName) {
final Tokenizer source = new StandardTokenizer();
TokenStream result = new StopFilter(source,
IrishAnalyzer.getDefaultHyphenations());
result = new ElisionFilter(result,
IrishAnalyzer.getDefaultElisonArticles());
// special
result = doMyFancyCustomStuff(result)
result = new IrishLowerCaseFilter(result);
result = new StopFilter(result,
IrishAnalyzer.getDefaultStopSet());
result = new SnowballFilter(result, new IrishStemmer());
return new TokenStreamComponents(source, result);
}
...but instead they have to read the source code for IrishAnalyzer and
copy/past the list of hyphenations & articles.
Do we want to change/standardize this?
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
