Re: Monitoring decisions taken by IndexOrDocValuesQuery
That should be doable and it is less invasive. I'll give it a try, Thanks! On Thu, Jun 10, 2021 at 2:21 PM Adrien Grand wrote: > This sounds useful indeed! > > Is there a way we could do it that wouldn't require forking > IndexOrDocValuesQuery? E.g. could we have query wrappers that we would use > on both the index query and the doc-value query in order to be able to > count how many times they have been used? We could add something like that > to lucene/sandbox. > > On Thu, Jun 10, 2021 at 2:51 PM Egor Moraru wrote: > > > Hi Adrien, > > > > In this specific use case our data is encoded as points and also is > stored > > as doc values. > > > > We use information about which execution path is chosen to decide > > if we can get away with storing this data only once and using one of > > the queries. > > > > On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand wrote: > > > > > FWIW a related PR was just merged that allows to introspect query > > > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's > > > different from your use-case though in that it is debugging information > > for > > > a single query rather than statistical information across lots of user > > > queries (and the approach on that other issue makes things much slower > so > > > you wouldn't like to enable it in production). > > > > > > Out of curiosity, what are you doing with this information about which > > > execution path is chosen? > > > > > > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru > > wrote: > > > > > > > Hi, > > > > > > > > At my current project we wanted to monitor for a specific field the > > > > fraction of indexed vs doc values queries executed by > > > > IndexOrDocValuesQuery. > > > > > > > > We ended up forking IndexOrDocValuesQuery and passing a listener that > > > > is notified when the query execution path is decided. > > > > > > > > Do you think this is something the community might be interested in? > > > > > > > > Kind regards, > > > > Egor Moraru. > > > > > > > > > > > > > -- > > > Adrien > > > > > > > > > -- > > > > Kind regards, > > Egor Moraru. > > > > > -- > Adrien > -- Kind regards, Egor Moraru.
Re: Monitoring decisions taken by IndexOrDocValuesQuery
This sounds useful indeed! Is there a way we could do it that wouldn't require forking IndexOrDocValuesQuery? E.g. could we have query wrappers that we would use on both the index query and the doc-value query in order to be able to count how many times they have been used? We could add something like that to lucene/sandbox. On Thu, Jun 10, 2021 at 2:51 PM Egor Moraru wrote: > Hi Adrien, > > In this specific use case our data is encoded as points and also is stored > as doc values. > > We use information about which execution path is chosen to decide > if we can get away with storing this data only once and using one of > the queries. > > On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand wrote: > > > FWIW a related PR was just merged that allows to introspect query > > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's > > different from your use-case though in that it is debugging information > for > > a single query rather than statistical information across lots of user > > queries (and the approach on that other issue makes things much slower so > > you wouldn't like to enable it in production). > > > > Out of curiosity, what are you doing with this information about which > > execution path is chosen? > > > > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru > wrote: > > > > > Hi, > > > > > > At my current project we wanted to monitor for a specific field the > > > fraction of indexed vs doc values queries executed by > > > IndexOrDocValuesQuery. > > > > > > We ended up forking IndexOrDocValuesQuery and passing a listener that > > > is notified when the query execution path is decided. > > > > > > Do you think this is something the community might be interested in? > > > > > > Kind regards, > > > Egor Moraru. > > > > > > > > > -- > > Adrien > > > > > -- > > Kind regards, > Egor Moraru. > -- Adrien
Re: Monitoring decisions taken by IndexOrDocValuesQuery
Hi Adrien, In this specific use case our data is encoded as points and also is stored as doc values. We use information about which execution path is chosen to decide if we can get away with storing this data only once and using one of the queries. On Wed, Jun 9, 2021 at 10:39 PM Adrien Grand wrote: > FWIW a related PR was just merged that allows to introspect query > execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's > different from your use-case though in that it is debugging information for > a single query rather than statistical information across lots of user > queries (and the approach on that other issue makes things much slower so > you wouldn't like to enable it in production). > > Out of curiosity, what are you doing with this information about which > execution path is chosen? > > On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru wrote: > > > Hi, > > > > At my current project we wanted to monitor for a specific field the > > fraction of indexed vs doc values queries executed by > > IndexOrDocValuesQuery. > > > > We ended up forking IndexOrDocValuesQuery and passing a listener that > > is notified when the query execution path is decided. > > > > Do you think this is something the community might be interested in? > > > > Kind regards, > > Egor Moraru. > > > > > -- > Adrien > -- Kind regards, Egor Moraru.
Re: Monitoring decisions taken by IndexOrDocValuesQuery
FWIW a related PR was just merged that allows to introspect query execution: https://issues.apache.org/jira/browse/LUCENE-9965. It's different from your use-case though in that it is debugging information for a single query rather than statistical information across lots of user queries (and the approach on that other issue makes things much slower so you wouldn't like to enable it in production). Out of curiosity, what are you doing with this information about which execution path is chosen? On Wed, Jun 9, 2021 at 2:14 PM Egor Moraru wrote: > Hi, > > At my current project we wanted to monitor for a specific field the > fraction of indexed vs doc values queries executed by > IndexOrDocValuesQuery. > > We ended up forking IndexOrDocValuesQuery and passing a listener that > is notified when the query execution path is decided. > > Do you think this is something the community might be interested in? > > Kind regards, > Egor Moraru. > -- Adrien
Monitoring decisions taken by IndexOrDocValuesQuery
Hi, At my current project we wanted to monitor for a specific field the fraction of indexed vs doc values queries executed by IndexOrDocValuesQuery. We ended up forking IndexOrDocValuesQuery and passing a listener that is notified when the query execution path is decided. Do you think this is something the community might be interested in? Kind regards, Egor Moraru.