[
https://issues.apache.org/jira/browse/LUCENE-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617272#comment-16617272
]
Adrien Grand commented on LUCENE-8501:
--------------------------------------
Storing floats as term frequencies not only raises issues with field length,
but also with storage efficiency since that will likely add 4 bytes for every
(term,document) pair of your dataset. For instance Lucene's FeatureField only
retains 16 bits of the 32 bits of floats: no sign bit, 8 exponent bits like
regular floats, and 8 mantissa bits instead of 23. It makes the overhead of
floats as term frequencies drop to 2 bytes per (term,document) pair instead of
4:
https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/core/src/java/org/apache/lucene/document/FeatureField.java#L153-L154
Would a similar approach work for you too?
> An ability to define the sum method for custom term frequencies
> ---------------------------------------------------------------
>
> Key: LUCENE-8501
> URL: https://issues.apache.org/jira/browse/LUCENE-8501
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Olli Kuonanoja
> Priority: Major
>
> Custom term frequencies allows expert users to index and score in custom
> ways, however, _DefaultIndexingChain_ adds a limitation to this as the sum of
> frequencies can't overflow
> {code:java}
> try {
> invertState.length = Math.addExact(invertState.length,
> invertState.termFreqAttribute.getTermFrequency());
> } catch (ArithmeticException ae) {
> throw new IllegalArgumentException("too many tokens for field \"" +
> field.name() + "\"");
> }
> {code}
> This might become an issue if for example the frequency data is encoded in a
> different way, say the specific scorer works with float frequencies.
> The sum method can be added to _TermFrequencyAttribute_ to get something like
> {code:java}
> invertState.length =
> invertState.termFreqAttribute.addFrequency(invertState.length);
> {code}
> so users may define the summing method and avoid the owerflow exceptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
