[
https://issues.apache.org/jira/browse/SOLR-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512346#comment-14512346
]
James Andres commented on SOLR-7461:
------------------------------------
Thanks for replying so quickly [~hossman].
What you say makes total sense. Calcdistinct is not very friendly to memory or
CPU on our Solr cluster, and we are aware of why! :)
Unfortunately, for me, being able to get exact (or very nearly exact) distinct
counts is important for the application I'm building (it's an analytics / stats
tool so correct numbers are a must have feature). The trouble is I rarely need
to know what the unique values actually are.
The specific itch I'm hoping to scratch is to avoid the 10MB to 30MB of
distinctValues coming over the wire to the app servers which my Python app
needs to parse then immediately discard. Without distinctValues my JSON
responses drop to a few kilobytes in size.
Regarding the hyperloglog approach:
- This is awesome. I can definitely use this :D
- Can you speculate on how the precision will be controlled? Will it be in the
query or in solrconfig.xml or both?
- Can I crank the precision way up such that when dealing with hundreds of
millions of unique items it will be nearly exact counts (eg: off by > 100)?
> StatsComponent, calcdistinct, ability to disable distinctValues while keeping
> countDistinct
> -------------------------------------------------------------------------------------------
>
> Key: SOLR-7461
> URL: https://issues.apache.org/jira/browse/SOLR-7461
> Project: Solr
> Issue Type: Improvement
> Reporter: James Andres
> Labels: statscomponent
>
> When using calcdistinct with large amounts of data the distinctValues field
> can be extremely large. In cases where the countDistinct is only required it
> would be helpful if the server did not return distinctValues in the response.
> I'm no expert, but here are some ideas for how the syntax could look.
> {code}
> # Both countDistinct and distinctValues are returned, along with all other
> stats
> stats.calcdistinct=true&stats.field=myfield
> # Only countDistinct and distinctValues are returned
> stats.calcdistinct=true&stats.field={!countDistinct=true
> distinctValues=true}myfield
> # Only countDistinct is returned
> stats.calcdistinct=true&stats.field={!countDistinct=true}myfield
> # Only distinctValues is returned
> stats.calcdistinct=true&stats.field={!distinctValues=true}myfield
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
