I could see Lucene help applications a bit more with building this sort of thing, e.g. by allowing applications to provide a listener to IndexWriterConfig that gets called on every segment deletion, flush, merge and live-docs/doc-values update with the total number of docs/bytes involved and the time it took. This should help applications track indexing-performance-related metrics more easily, e.g. write amplification through segment merges.
On Wed, Mar 19, 2025 at 1:22 PM Michael Sokolov <msoko...@gmail.com> wrote: > One idea I've heard batted around is to override > IndexSearcher.createWeight in a profiling IndexSearcher and then wrap > Weights and finally Scorers in order to emit metrics on every advance > or other low-level operation. This could help with search profiling I > guess although of course it will slow everything down, it might be > helpful for counting operations. But maybe your problem is during > indexing? > > On Wed, Mar 19, 2025 at 5:59 AM Robert Muir <rcm...@gmail.com> wrote: > > > > opentelemetry is anything but lightweight, not recommended. Don't > > waste your time with it. if you want to expose 'metrics', it is really > > up to the application to do that, lucene is just a library. I'd > > recommend just exposing via HTTP and using prometheus & co. > > > > Debugging performance issues in the library is totally different and > > telemetry as a word doesn't make sense for that You can setup 'perf' > > to debug performance issues. You can also run with lightweight JFR > > profiling built into the JVM. The nightly benchmarks already do that. > > > > On Wed, Mar 19, 2025 at 2:16 AM Vigya Sharma <vigya.w...@gmail.com> > wrote: > > > > > > Hi All, > > > > > > I am debugging an unexpected throughput regression in my feature > branch which has changes across multiple files and functions, and was > wondering if there's a better way to identify the slowdown than adding > elapsed time logs to infostream. > > > > > > Do we have support in Lucene for telemetry on internal operations? It > would be nice to be able to add timers, counters, and metrics, to identify > performance changes. This would hide disabled behind a gradle flag. We > could enable it for benchmark runs and possibly chart these metrics in > nightly runs. > > > > > > Has this been discussed before? I'm happy to explore this more if it > sounds like a good idea. Any suggestions for lightweight, open-source > telemetry libraries that I should look into? I'm considering OpenTelemetry > but open to alternate suggestions and ideas. > > > > > > Vigya > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- Adrien
