[
https://issues.apache.org/jira/browse/SOLR-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199726#comment-17199726
]
Gus Heck commented on SOLR-14597:
---------------------------------
looks like LUCENE-9531 has caused a conflict with the patch, and there have
been some changes to the gradle files running javacc so I'm working on updating
to work with that and I'll publish the fix as a pull-request for easier review.
Now that there is code to look at some responses:
[~arafalov]: Point noted about TypeTokenFilter, there is similarity though
filtering on flags instead of types. It would be attractive to also inherit
from FilteringTokenFilter but It looks like one edge case I ran into isn't
handled by the super class. (and makes me wonder if there's a lurking issue
with other FilteringTokenFilter sub classes. The case I ran into is thus: The
first token in the stream gets assigned a synonym, then in a subsequent step
the first token is dropped (this is quite intentional in some use cases we had
where the intent was to entirely prevent matches on the original token, but
still match on the synonym). When this happens it causes
{{java.lang.IllegalArgumentException: first position increment must be > 0 (got
0)}} despite the fact that this scenario is not actually an error in terms of
which tokens we want. Unfortunately there's no good way to know what's going to
happen to the next token (which may not have the flags in question) so I came
up with a workaround that I'm not very pleased with dropping in a placeholder
token that is unlikely to match anything. Open to suggestions for better
options there, and interested in whether or not other filters that drop tokens
can hit the same issue, or if they've handled it in some graceful way I'm not
appreciating.
Also, now that the code is available, let me know if you still see similarity
between PatternTypingFilterFactory and KeywordMarkerFilterFactory... I think
they are quite different.
[~ichattopadhyaya], [~dsmiley] While some of this could potentially be broken
out into a package, there are also some changes to core and some lucene level
classes that probably wouldn't want to be in a package, so feel free to put
some eyes on it and suggest what the dividing line is (more eyes == better).
I'm not against the idea of a 1st party package, but the question is will this
be popular enough to merit default inclusion? Another breaking new ground sort
of question is "Is it easier to pull it in later or push it out to a package
later if we change our minds?" Maybe neither is harder...
Changes to note to classes outside the new org.apache.solr.aqp package (where
the meat of the new parser and it's .jj file lives):
# TypeAsSynonymFilter is gaining the ability to manage what flags are
transmitted from the original token to the synonym when it is created
# BaseTokenStreamTestCase is gaining the ability to verify the flags on the
tokens produced.
# access org.apache.solr.cloud.AbstractDistribZkTestBase#copyConfigUp is
opened up so that it can be used in a wider array of tests.
# Solr gains TokenAnalyzerFilter which applies the Analyzer from a specified
field type to the individual tokens of the current stream (see javadoc for more
detail)
# Operator and SynonymQueryStyle are extracted from the standard parser's base
class so they can be re-used. Reuse is is necessary because TextField
references SynonymQueryStyle directly.
# The above change forces an compile time API change in TextField, which might
force this to not be available till 9.x (though the desire to make AQP
available in 8.x is there).
# The change to TextField then failed TestPackages which failed with a
ClassNotFound when it went looking for the old SynonymeQueryStyle inner class
that had been promoted to a separate class. This forced me to decompile and
provide classes and build/rebuild support for the binary jars checked in for
TestPackages (as *.jar.bin). (the .java files for the classes loaded by this
test had not been checked in). This is the genesis of the o.a.smy.pkg package
namespace.
Some of the above (especially #7) might want to be broken into related or
sub-tickets.
> Advanced Query Parser
> ---------------------
>
> Key: SOLR-14597
> URL: https://issues.apache.org/jira/browse/SOLR-14597
> Project: Solr
> Issue Type: New Feature
> Components: query parsers
> Reporter: Mike Nibeck
> Assignee: Gus Heck
> Priority: Major
> Attachments: aqp_patch.patch
>
>
> This JIRA ticket tracks the progress of SIP-9, the Advanced Query Parser that
> is being donated by the Library of Congress. Full description of the feature
> can be found on the SIP Page.
> [https://cwiki.apache.org/confluence/display/SOLR/SIP-9+Advanced+Query+Parser]
> Briefly, this parser provides a comprehensive syntax for users that use
> search on a daily basis. It also reserves a smaller set of punctuators than
> other parsers. This facilitates easier handling of acronyms and punctuated
> patterns with meaning ( such as C++ or 401(k) ). The new syntax opens up some
> advanced features while also preventing access to arbitrary features via
> local parameters. This parser will be safe for accepting user queries
> directly with minimal pre-parsing, but for use cases beyond it's established
> features alternate query paths (using other parsers) will need to be supplied.
> The code drop is being prepared and will be supplied as soon as we receive
> guidance from the PMC regarding the proper process. Given that the Library
> already has a signed CCLA we need to understand which of these (or other
> processes) apply:
> [http://incubator.apache.org/ip-clearance/ip-clearance-template.html]
> and
> [https://www.apache.org/licenses/contributor-agreements.html#grants]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
