[
https://issues.apache.org/jira/browse/SOLR-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18143247#comment-18143247
]
Nick Shanin edited comment on SOLR-18616 at 11/11/26, 10:04 PM:
----------------------------------------------------------------
Hi Jan, Chris,
I'm willing to work on this, I could do all the steps that Jan mentioned. Let
me know if this is a good first step.
*What happens today*
Solr 10 no longer exposes the Jetty metrics that previously showed HTTP
requests and responses outside a valid Solr collection or core, or the
utilization of Jetty's request thread pool. For example, a spike of PUT
requests to a nonexistent path is not visible in the existing Solr metrics.
*What this change does*
I tried a server-owned Jetty handler that counts completed requests by
method and responses by status class, including requests outside the Solr
context. It also reports active requests and Jetty thread-pool current,
idle, busy, maximum, and queued counts. Busy threads include leased threads
and other Jetty work, so they are distinct from active HTTP requests.
The initialization sequence addresses the issue Chris described:
# Jetty creates the handler before loading the Solr webapp. The handler
records counters independently and does not need SolrMetricManager.
# Jetty makes the handler available to the webapp through a servlet-context
attribute.
# After CoreContainer creates SolrMetricManager, Solr registers observable
instruments that read the handler's values. The registration is closed
with the webapp's metrics context.
The candidate changes Jetty configuration and packaging, adds the handler
module, and connects it to CoreContainerProvider. It also adds tests,
documentation, and a changelog entry.
*Proof*
On Linux at {{{}2a5127ff30608f7ea0248277df83c5daea4d1927{}}}:
* {{JettyMetricsBridgePrototypeTest}} passed 2/2 tests.
* A packaged Solr distribution counted one outside-context PUT exactly once
and recorded the expected 4xx responses for it and an invalid-collection
request. The thread-pool gauges were present.
* Four additional scratch probes passed for held requests, failure,
transport-abort handling, and concurrent requests. Metrics-disabled
startup, restart, and an embedded container without the bridge also
behaved as expected.
* The distribution build, tidy, and root {{check -x test}} passed.
The additional probes are not yet committed regression tests. Their test
code was preserved for review and potential inclusion.
*A choice to check*
Jan recommended starting with Jetty's {{StatisticsHandler}} and
{{{}ConnectionStatistics{}}}. That would avoid a custom handler module and
provide aggregate request, response, connection, byte, and thread-pool
information.
I tried the custom-handler option because the ticket's PUT-spike example
seems to call for a per-method breakdown, which {{StatisticsHandler}} does
not provide. In the Jetty version examined, several cumulative
{{StatisticsHandler}} getters also return {{int}} values, so a direct
counter bridge would need to account for possible wraparound on busy,
long-running nodes.
The tradeoff is more Solr-owned code: a handler module, packaging, and a
handoff across the server and webapp boundary. If aggregate traffic counts
are sufficient for a first step, Jan's proposed approach is simpler.
My recommendation is to retain per-method counts for the reported PUT
case. I am not opposed to the built-in handlers: if preserving more of the
previous visibility is important, they may complement the custom handler.
That combination would need a closer look at overlapping metrics, counter
semantics, and how to avoid exposing the same measurement twice.
Is per-method visibility worth the custom handler here, or would you
prefer the built-in handlers as the first step?
*Limits*
This candidate addresses the ticket's two stated visibility gaps; it is
not a full replacement for the former Dropwizard instrumentation.
* It reports request counts by method and response counts by status
class, but not request-duration distributions, dispatch or suspended
request metrics, async-specific counts, or the old decayed rates and
error-rate ratios.
* It does not add the connection and byte statistics in Jan's proposed
{{StatisticsHandler}} and {{ConnectionStatistics}} approach.
* Busy threads are current minus idle, including leased threads and Jetty
work unrelated to HTTP requests. They are not a count of active HTTP
requests; the candidate reports those separately.
* A transport-aborted request whose completion callback fails is counted
as a 5xx. The production servlet-layer async-abort path was not exercised.
* The additional async, concurrency, and lifecycle probes passed as
scratch tests, but are not yet committed regression tests.
*Appendix: built-in and custom handlers*
|| ||Built-in Jetty handlers||Custom handler in this candidate||
|Instrumentation|Jetty's {{{}StatisticsHandler{}}}, optionally paired with
{{ConnectionStatistics}}|A new Solr-owned {{JettyMetricsHandler}}|
|Request data|Aggregate request counts, response classes, timing, and other
statistics|Counts by HTTP method and response class, plus active requests|
|Distinguishes PUT from GET|No|Yes|
|Connection and byte data|Available through the proposed built-in
combination|Not included|
|Solr code needed|A bridge that reads Jetty's statistics and registers them
with {{SolrMetricManager}}|The handler and the bridge|
|Main advantage|Less custom instrumentation to maintain|Covers the ticket's
per-method PUT-spike example|
Both approaches need the late bridge: Jetty starts before
{{SolrMetricManager}} exists. The Jetty-side component collects data
first, and Solr registers instruments to read it after the webapp starts.
[Candidate diff against its Apache main
base|https://github.com/nick-boss-tech/solr/compare/82888694ae248b74d105d9c7b5f60d33c385f325...2a5127ff30608f7ea0248277df83c5daea4d1927]
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
was (Author: JIRAUSER314749):
Hi Jan, Chris
*What happens today*
Solr 10 no longer exposes the Jetty metrics that previously showed HTTP
requests and responses outside a valid Solr collection or core, or the
utilization of Jetty's request thread pool. For example, a spike of PUT
requests to a nonexistent path is not visible in the existing Solr metrics.
*What this change does*
I tried a server-owned Jetty handler that counts completed requests by
method and responses by status class, including requests outside the Solr
context. It also reports active requests and Jetty thread-pool current,
idle, busy, maximum, and queued counts. Busy threads include leased threads
and other Jetty work, so they are distinct from active HTTP requests.
The initialization sequence addresses the issue Chris described:
# Jetty creates the handler before loading the Solr webapp. The handler
records counters independently and does not need SolrMetricManager.
# Jetty makes the handler available to the webapp through a servlet-context
attribute.
# After CoreContainer creates SolrMetricManager, Solr registers observable
instruments that read the handler's values. The registration is closed
with the webapp's metrics context.
The candidate changes Jetty configuration and packaging, adds the handler
module, and connects it to CoreContainerProvider. It also adds tests,
documentation, and a changelog entry.
*Proof*
On Linux at {{2a5127ff30608f7ea0248277df83c5daea4d1927}}:
* {{JettyMetricsBridgePrototypeTest}} passed 2/2 tests.
* A packaged Solr distribution counted one outside-context PUT exactly once
and recorded the expected 4xx responses for it and an invalid-collection
request. The thread-pool gauges were present.
* Four additional scratch probes passed for held requests, failure,
transport-abort handling, and concurrent requests. Metrics-disabled
startup, restart, and an embedded container without the bridge also
behaved as expected.
* The distribution build, tidy, and root {{check -x test}} passed.
The additional probes are not yet committed regression tests. Their test
code was preserved for review and potential inclusion.
*A choice to check*
Jan recommended starting with Jetty's {{StatisticsHandler}} and
{{ConnectionStatistics}}. That would avoid a custom handler module and
provide aggregate request, response, connection, byte, and thread-pool
information.
I tried the custom-handler option because the ticket's PUT-spike example
seems to call for a per-method breakdown, which {{StatisticsHandler}} does
not provide. In the Jetty version examined, several cumulative
{{StatisticsHandler}} getters also return {{int}} values, so a direct
counter bridge would need to account for possible wraparound on busy,
long-running nodes.
The tradeoff is more Solr-owned code: a handler module, packaging, and a
handoff across the server and webapp boundary. If aggregate traffic counts
are sufficient for a first step, Jan's proposed approach is simpler.
My recommendation is to retain per-method counts for the reported PUT
case. I am not opposed to the built-in handlers: if preserving more of the
previous visibility is important, they may complement the custom handler.
That combination would need a closer look at overlapping metrics, counter
semantics, and how to avoid exposing the same measurement twice.
Is per-method visibility worth the custom handler here, or would you
prefer the built-in handlers as the first step?
*Limits*
This candidate addresses the ticket's two stated visibility gaps; it is
not a full replacement for the former Dropwizard instrumentation.
* It reports request counts by method and response counts by status
class, but not request-duration distributions, dispatch or suspended
request metrics, async-specific counts, or the old decayed rates and
error-rate ratios.
* It does not add the connection and byte statistics in Jan's proposed
{{StatisticsHandler}} and {{ConnectionStatistics}} approach.
* Busy threads are current minus idle, including leased threads and Jetty
work unrelated to HTTP requests. They are not a count of active HTTP
requests; the candidate reports those separately.
* A transport-aborted request whose completion callback fails is counted
as a 5xx. The production servlet-layer async-abort path was not exercised.
* The additional async, concurrency, and lifecycle probes passed as
scratch tests, but are not yet committed regression tests.
*Appendix: built-in and custom handlers*
|| || Built-in Jetty handlers || Custom handler in this candidate ||
| Instrumentation | Jetty's {{StatisticsHandler}}, optionally paired with
{{ConnectionStatistics}} | A new Solr-owned {{JettyMetricsHandler}} |
| Request data | Aggregate request counts, response classes, timing, and other
statistics | Counts by HTTP method and response class, plus active requests |
| Distinguishes PUT from GET | No | Yes |
| Connection and byte data | Available through the proposed built-in
combination | Not included |
| Solr code needed | A bridge that reads Jetty's statistics and registers them
with {{SolrMetricManager}} | The handler and the bridge |
| Main advantage | Less custom instrumentation to maintain | Covers the
ticket's per-method PUT-spike example |
Both approaches need the late bridge: Jetty starts before
{{SolrMetricManager}} exists. The Jetty-side component collects data
first, and Solr registers instruments to read it after the webapp starts.
[Candidate diff against its Apache main
base|https://github.com/nick-boss-tech/solr/compare/82888694ae248b74d105d9c7b5f60d33c385f325...2a5127ff30608f7ea0248277df83c5daea4d1927]
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
> Solr 10.+ no longer provides any jetty metrics
> ----------------------------------------------
>
> Key: SOLR-18616
> URL: https://issues.apache.org/jira/browse/SOLR-18616
> Project: Solr
> Issue Type: Sub-task
> Reporter: Chris M. Hostetter
> Priority: Major
>
> [https://lists.apache.org/thread.html/8yzggfvrohv09pqs2wgq1y1nfsq80xhc]
>
> Prior to Solr 10, Solr leveraged 2 "dropwizard" provided jetty plugins in
> order to report some key jetty metrics...
> * InstrumentedEE10Handler
> * InstrumentedQueuedThreadPool
> Those were removed by SOLR-17855, w/o any replacement:
> [https://github.com/apache/solr/commit/cee53df8e0a]
>
> The major impacts of this loss are:
> # no other (existing) solr metrics provide visibility into the number of
> HTTP requests, and their associated responses that do *NOT* a valid Solr
> collection/core
> ** so for example: there is no indication in the existing 10.x metrics of a
> huge spike of abnormal {{PUT}} requests to a patch that doesn't exist.
> # no other (existing) solr metrics provide visibility into the thread
> utilization of the jetty (request) thread pool
> ** so for example: there is no indication in the existing 10.x metrics that
> the current number of currently executing requests is growing continuously
> and getting close to saturating the max number of threads
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
