Thomas Mueller created OAK-9811:
-----------------------------------
Summary: Statistics index
Key: OAK-9811
URL: https://issues.apache.org/jira/browse/OAK-9811
Project: Jackrabbit Oak
Issue Type: Improvement
Components: indexing, query
Reporter: Thomas Mueller
Assignee: Thomas Mueller
Queries should be as fast as possible:
* They should read as little data as possible (low I/O)
* Network roundtrips should be reduced (see also OAK-9780)
* In-memory processing should be fast (low CPU usage)
To do that:
* Queries needs to _have_ the right indexes. Possibly indexes need to be added
(which might be a manual task, or semi-automated, or fully automated). For a
developer, it would also be good to know how fast a query could be, if an index
is added.
* Queries should _use_ the right indexes. Sometimes multiple indexes can be
used.
* Queries should use the right execution plan (for example: a join can be
evaluated in multiple ways).
For this, it is great to have accurate statistics. We currently have statistics
about number of nodes per path (approximate counter), and document statistics
for Lucene and Elastic indexes.
But we don't have statistics for _unindexed_ data currently. That would be good
to have: which property (by property name) is how common? How many distinct
values are there per property? What is the histogram? And so on. For this,
something like the counter index could be added, that is updated using a
streaming algorithm. We need to ensure the number of writes to this index is
low (e.g. less than 1% of the overall writes), and memory usage is very low.
There are a number of such libraries, but arguably we could implement this
ourselves, as our use case is untypical (reduced number of writes, reduced
memory usage). https://github.com/thomasmueller/tinyStats and related libraries
could be used as a starting point.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
