Hi all,
I am quite new to the Lucene world and recently started using its python
wrapper (PyLucence) in my project.
So far, I have been using the token based querying method which works fine.
But, now I want to modify the querying approach as the following:
- Given the query string
- extract all its terms (n-grams; n>=2)
- for each term search the Indexer and return the (k) documents which
contain that specific term
let's say that the input text is: "search for this !"
And I want to search for each of these sub-strings, separately:
"search for", "for this", "this !", "search for this", "for this ."
I tried with Shingle, but I couldn't get the desired output. You can find
my code at the end.
I added line 14 to the code to avoid searching for unigrams which are of no
interest to me.
It is worth mentioning that for creating both the Indexer and Searcher I
used the following settings:
analyzer = ShingleAnalyzerWrapper(WhitespaceAnalyzer(), 2, 4, ' ', True,
False, None)
The point is that if I remove line 14, it does return me some documents
which they contain the words of the given n-gram, but they do not
necessarily make one single phrase. I mean if the n-gram is *"search for
this"* it might return a document like: *"please search for that and
this"*
Which means it just looked for the unigrams, not the whole string.
Any idea about this issue?
Thanks,
Amin
1 def query(self, queryString):
2 vec = {}
3 idx = 0
4 ret_documents = {}
5 analyzer = ShingleAnalyzerWrapper(WhitespaceAnalyzer(), 2, 4, ' ',
True, False, None)
6 ts = self.analyzer.tokenStream("source", StringReader(queryString))
7 termAtt = ts.addAttribute(CharTermAttribute.class_)
8 ts.reset()
9
10 all_grams = []
11
12 while ts.incrementToken():
13 ngram = termAtt.toString()
14 if len(ngram.split()) > 1:
15 all_grams.append(ngram)
16 ts.close()
17
18 for ngram in all_grams:
19 query = BooleanQuery.Builder()
20 query.add(TermQuery(Term("source", ngram)),
BooleanClause.Occur.MUST)
21 scoreDocs = self.searcher.search(query.build(),
self.max_retriever).scoreDocs
22
23 for scoreDoc in scoreDocs:
24 doc = self.searcher.doc(scoreDoc.doc)
25 ret_documents[idx] = [doc.get("id"), scoreDoc.score,
doc.get("source")]
26 idx += 1
27
28 print "RET_DOCS: ", ret_documents
