Re: Total number of terms in index for collection frequency

[Dirk Rothe]

Hi Floran,

we're looping over all lucene-docs, apply the appropriate analyzer,  
iterate and collect the distinct tokens. Pretty inefficient I guess, but  
you also get the frequency for each unique token. Nice for checking:  
https://en.wikipedia.org/wiki/Zipf%27s_law

--dirk

Am 18.08.2016, 16:31 Uhr, schrieb Gmehlin  Floran  
<fgmeh...@student.ethz.ch>:

Hi,

I am struggling to compute the collection frequency of a term (PyLucene  
4.10.1).
So far, I can have the collection count of terms with :

reader = IndexReader.open(SimpleFSDirectory(File(LUCENE_INDEX)))
termVector = reader.getTermVector(docID, "contents");
termsEnumvar = termVector.iterator(None)
termsref = BytesRefIterator.cast_(termsEnumvar)
cf_dict = {}
try:
    while (termsref.next()):
        termval = TermsEnum.cast_(termsref)
        fg = termval.term().utf8ToString()
        cf = reader.totalTermFreq(Term("contents", termval.term())    #  
collection count
        cf_dict[fg]=cf
except StopIteration, e:
    print ''

I would like to have the "frequency" in cf_dict instead of the count.  
For this, I need to divide it with the total number of indistinct terms  
in the index.

Does anyone know how to get this ?

Thank you for your help,

Floran