To be clear, this is a blocker for Solr 10. Shipping without this would mean waiting till Solr 11. It would *suck* to wait that long, and it'd suck for everyone maintaining the main branch & branch_10x branch with this significant difference. We'll get this transition done.
On Wed, Aug 13, 2025 at 9:37 PM Matthew Biscocho (BLOOMBERG/ 919 3RD A) < mbisco...@bloomberg.net> wrote: > Hi everyone, > > I wanted to share an update on the progress of the OTEL (Open Telemetry) > migration from Dropwizard. If you haven’t been following, most of the work > is being tracked under SOLR-17458 and merged into the feature/SOLR-17458 > branch. The foundation for OTEL instrumentation in Solr is mostly in place > with metrics having attributes/labels, along with some removals to slim > down the project. > > Here are some highlights: > */admin/metrics now exposes Prometheus-formatted metrics, with OpenMetrics > format available as an optional parameter. > *JMX, Graphite, and SLF4J reporters have been removed. > *Prometheus Exporter removed (currently in PR review) > *OTLP metric exporter to push metrics is included in the OpenTelemetry > module (currently in PR review). > > The /admin/metrics endpoint with prometheus and OTLP metric exporter > specifically opens Solr to a lot of new supported options of monitoring > with different backends and pipelines. > > I have also migrated a good chunk of metrics over to OTEL but there are > still a handful of components in progress. Still aiming for Solr 10 release > and hopeful we will get there. If anyone is interested in contributing or > helping out, I have a list of additional metrics still needing migration to > OTEL. Also big thanks to David for reviewing almost all the code changes > and helping with this project! > > - Matt > > From: dev@solr.apache.org At: 05/13/25 15:19:26 UTC-4:00To: > dev@solr.apache.org > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry > > Sorry for not following up in a timely manner; I was on vacation, plus > personal stuff, and now/soon changed jobs. I mostly just want to connect > to improve the SIP then identify next steps. Just DM me to connect at your > next convenience. The outcome will be a better SIP and we share it again. > > On Mon, Apr 21, 2025 at 10:42 AM Matthew Biscocho (BLOOMBERG/ 919 3RD A) < > mbisco...@bloomberg.net> wrote: > > > Happy to schedule an open video for anyone who wants to join for more > > discussion around the SIP and improve on it. The more the better. Also > with > > this over haul, might be worth adding additional metrics or events to > > measure. For example, Solr's prometheus exporter gets live nodes and > shard > > state metrics by pulling the GET `/admin/collections CLUSTERSTATUS` API. > So > > it's something missing directly from the metrics api. Might be worth > > measuring the event with a gauge instead of it having to call and pull > from > > a separate API. > > > > As for OTEL adoption, seems like this is becoming common. Spring Boot > uses > > Micrometer for it's metrics framework which itself has a > OtlpMeterRegistry > > > > https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micro > > meter-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry > <https://github.com/micrometer-metrics/micrometer/blob/main/implementations/micrometer-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry> > . > java. > > Open Telemetry has a large instrumentation list of support libraries for > > its java agent > > > > https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/d > ocs/supported-libraries.md#libraries--frameworks > <https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks> > > including a spring boot one that works. > > > > From: dev@solr.apache.org At: 04/18/25 18:07:39 UTC-4:00To: > > dev@solr.apache.org > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry > > > > It's notable that Solr-core in main already includes OpenTelemetry > > dependencies. I expect this will be increasingly common, like SLF4J has > > become. > > > > Docker: definitely a separate topic altogether. See > > https://issues.apache.org/jira/browse/SOLR-17653 which I'm likely to > > pursue. I'm very familiar with running Solr in Docker in integration > > tests. > > > > On Fri, Apr 18, 2025 at 10:47 AM Gus Heck <gus.h...@gmail.com> wrote: > > > > > This sounds good to me because Dropwizard metrics is used by the > > Dropwizard > > > framework. Recent experiences where Dropwizard is used as an > intermediary > > > layer and integration tests using our test framework have shown it's a > > path > > > to jar hell with Solr wanting one version and the Dropwizard based code > > > wanting another... What I don't know is the extent to which OTel might > be > > > adopted by other similar frameworks (what does spring boot usually work > > > with?) The issue of us having several hundred dependencies and folks > > using > > > our test framework is always going to have some clash but maybe this > will > > > help? > > > > > > Side topic: I'm aware of at least two places that have come to the > > > conclusion that integrated tests should spin up a separate JVM for > solr, > > > and part of me wonders if there's a way to push our test framework to > do > > > that natively so that folks using solr don't need to do a lot of > > > engineering to create integration tests. This might also benefit our > > tests > > > by having a separate log for the server and the client side of > > > interactions. Currently, you can't just scroll through and look for the > > > stack trace trivially because often you have one reported from the > server > > > and one reported from the test itself. We might even be able to set a > "no > > > stack trace in the logs" general standard for the server side of things > > > this way and fail any tests that generate a stack trace to ensure that > a > > > healthy solr never logs stack traces. Such a thing might involve > having a > > > testing module (not loaded in production) that allows/simplifies > > > configuration without security hurdles... > > > > > > The obvious question of course is speed.... But maybe a solidification > of > > > what's an integration test and what's a unit test helps there, and > > > integration tests are intended for build infra, or final checks rather > > than > > > developer testing during development. > > > > > > -Gus > > > > > > On Thu, Apr 17, 2025 at 11:45 PM David Smiley <dsmi...@apache.org> > > wrote: > > > > > > > That's a nice start! I'd like to work with you on making the SIP > > better. > > > > Maybe a 1:1 (others invited too!) video might be most productive. > I'd > > > love > > > > to involve AB more here. I think the SIP template that you clearly > > > started > > > > from isn't great; I'm tempted to replace it with a list of > > considerations > > > > and not a proposed format. People should structure their SIP however > > the > > > > topic best presents itself. > > > > > > > > On Thu, Apr 17, 2025 at 2:34 PM Matthew Biscocho (BLOOMBERG/ 919 3RD > > A) < > > > > mbisco...@bloomberg.net> wrote: > > > > > > > > > Based on the discussion from yesterdays meetup, created SIP-23 for > > > moving > > > > > to Open Telemetry > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/SOLR/SIP-23%3A+Switch+from+DropWizar > > d+to+OpenTelemetry > > > > > > > > > > With Solr 10 around the corner, hoping this work can get done > before > > > > then. > > > > > > > > > > From: dev@solr.apache.org At: 04/14/25 01:04:13 UTC-4:00To: > > > > > dev@solr.apache.org > > > > > Subject: Re: SIP proposal: Switch from DropWizard to OpenTelemetry > > > > > > > > > > I'm excited about the change! Where I work, it would > *significantly* > > > > > simplify our metrics pipeline if Solr were to embrace OTel for > > metrics, > > > > as > > > > > we could then use company-provided OTel plugins. The broad > industry > > > > > adoption of OTel points to this being the least friction. With > > > > DropWizard, > > > > > it appears we hacked attributes onto it, in a sense. > > > > > > > > > > The primary criteria/requirement that comes to mind is to have > > > > > strong/sophisticated ways to filter the right metrics to publish. > > Solr > > > > has > > > > > that today. > > > > > > > > > > On Fri, Apr 11, 2025 at 5:00 PM Matthew Biscocho (BLOOMBERG/ 919 > 3RD > > > A) < > > > > > mbisco...@bloomberg.net> wrote: > > > > > > > > > > > Hey everyone, > > > > > > > > > > > > Making this thread because I was interested in writing up a SIP > for > > > > > > SOLR-17458 https://issues.apache.org/jira/browse/SOLR-17458 and > > > > starting > > > > > > a discussion around this. The proposed change involves migrating > > > Solr's > > > > > > metrics framework from DropWizard to OpenTelemetry (OTel). This > > will > > > > move > > > > > > Solr to an attribute based metric framework and will also help > > > > > > future-proofing by keeping it from vendor lock-in, given > > > > OpenTelemetry's > > > > > > stance on being vendor-neutral and working with many different > > > > pipelines. > > > > > > Since Solr already packages an Open Telemetry SDK as a module for > > > > > exporting > > > > > > spans through OTLP, we can use that foundation to have it push > > > metrics > > > > as > > > > > > well. There will be a significant change here which will probably > > > break > > > > > > many components and no longer be backwards compatible so this > will > > > > > probably > > > > > > need to be a major release version change but we can potentially > > > keep a > > > > > few > > > > > > things backwards compatible if needed. Just some things off the > top > > > of > > > > my > > > > > > head: > > > > > > > > > > > > * The Prometheus Exporter (Maybe should be deprecated?) > > > > > > * GET /admin/metrics endpoint > > > > > > * JVM metrics collected from OTel > > > > > > * Metric reporters? > > > > > > > > > > > > Anyone have thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > http://www.needhamsoftware.com (work) > > > https://a.co/d/b2sZLD9 (my fantasy fiction book) > > > > > > > > > > > >
