[jira] [Updated] (LUCENE-8884) Add Directory wrapper to track per-query IO counters
[
https://issues.apache.org/jira/browse/LUCENE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-8884:
---
Attachment: LUCENE-8884.patch
Status: Open (was: Open)
Another iteration folding [~rcmuir]'s feedback.
I was worried that the thread that called {{clone()}} may not be the same
thread that then consumes the {{IndexInput}} and added an assertion, but it
looks like it's OK. I added another random test too, and improved javadocs.
I could not eliminate the required {{setKeyForThread}} call because even in the
single threaded case, where only one thread executes the query across all
segments, the directory wrapper still needs to know which query that is to
track its IO counters.
I haven't tested performance impact of this but it's likely minor now since we
now retrieve the counters on {{clone()}} instead of on every IO operation.
> Add Directory wrapper to track per-query IO counters
>
>
> Key: LUCENE-8884
> URL: https://issues.apache.org/jira/browse/LUCENE-8884
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
> Attachments: LUCENE-8884.patch, LUCENE-8884.patch
>
>
> Lucene's IO abstractions ({{Directory, IndexInput/Output}}) make it really
> easy to track counters of how many IOPs and net bytes are read for each
> query, which is a useful metric to track/aggregate/alarm on in production or
> dev benchmarks.
> At my day job we use these wrappers in our nightly benchmarks to catch any
> accidental performance regressions.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8884) Add Directory wrapper to track per-query IO counters
[
https://issues.apache.org/jira/browse/LUCENE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-8884:
---
Attachment: LUCENE-8884.patch
Status: Open (was: Open)
Trying again to attach first cut patch!
> Add Directory wrapper to track per-query IO counters
>
>
> Key: LUCENE-8884
> URL: https://issues.apache.org/jira/browse/LUCENE-8884
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
> Attachments: LUCENE-8884.patch
>
>
> Lucene's IO abstractions ({{Directory, IndexInput/Output}}) make it really
> easy to track counters of how many IOPs and net bytes are read for each
> query, which is a useful metric to track/aggregate/alarm on in production or
> dev benchmarks.
> At my day job we use these wrappers in our nightly benchmarks to catch any
> accidental performance regressions.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8884) Add Directory wrapper to track per-query IO counters
[
https://issues.apache.org/jira/browse/LUCENE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-8884:
---
Status: Open (was: Open)
Here's an initial patch, adding {{IOTrackingDirectoryWrapper.}}
Whenever a given thread is "working" on a particular query it must first call
{{setQueryForThread}} so the wrapper knows which query's counters to increment.
It tracks number of IOPs and how many total bytes were read.
It's likely it impacts search performance, so it should only be used during
profiling/benchmarking.
> Add Directory wrapper to track per-query IO counters
>
>
> Key: LUCENE-8884
> URL: https://issues.apache.org/jira/browse/LUCENE-8884
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/store
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Minor
>
> Lucene's IO abstractions ({{Directory, IndexInput/Output}}) make it really
> easy to track counters of how many IOPs and net bytes are read for each
> query, which is a useful metric to track/aggregate/alarm on in production or
> dev benchmarks.
> At my day job we use these wrappers in our nightly benchmarks to catch any
> accidental performance regressions.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
