Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy merged PR #3908: URL: https://github.com/apache/solr/pull/3908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
dsmiley commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2587170219
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -136,8 +125,15 @@ private static void handleNamedListFormat(
/** Makes a remote request asynchronously. */
public static CompletableFuture> callRemoteNode(
- String nodeName, String uriPath, SolrParams params, ZkController
zkController)
- throws IOException, SolrServerException {
+ String nodeName, String uriPath, SolrParams params, ZkController
zkController) {
+
+// Validate that the node exists in the cluster
+if
(!zkController.zkStateReader.getClusterState().getLiveNodes().contains(nodeName))
{
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Requested node " + nodeName + " is not part of cluster");
Review Comment:
I was tempted to leave the check in `resolveNodes` but use `resolveNodes`
for both code paths. But `resolveNodes` references a specific parameter
`nodes`. It could be generalized a bit.
I'm fine leaving it or doing that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2586497596
##
solr/core/src/java/org/apache/solr/response/PrometheusResponseWriter.java:
##
@@ -41,7 +43,26 @@ public void write(
OutputStream out, SolrQueryRequest request, SolrQueryResponse response,
String contentType)
throws IOException {
+// Check if we have pre-formatted Prometheus text (from single-node proxy)
+if (response.getValues().get("stream") instanceof InputStream stream) {
+ try {
+stream.transferTo(out);
+ } finally {
+stream.close();
+ }
+ return;
+}
+
+if (response.getException() != null) {
+
out.write(response.getException().toString().getBytes(StandardCharsets.UTF_8));
+ return;
Review Comment:
Well spotted
##
changelog/unreleased/SOLR-18004-frontend-parse-prometheus.yml:
##
@@ -4,6 +4,7 @@ type: fixed # added, changed, fixed, deprecated, removed,
dependency_update, sec
authors:
- name: Jan Høydahl
url: https://home.apache.org/phonebook.html?uid=janhoy
+ - name: David Smiley
Review Comment:
Satisfying to see logchange in actuve use, with no risk of merge conflicts!
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -136,8 +125,15 @@ private static void handleNamedListFormat(
/** Makes a remote request asynchronously. */
public static CompletableFuture> callRemoteNode(
- String nodeName, String uriPath, SolrParams params, ZkController
zkController)
- throws IOException, SolrServerException {
+ String nodeName, String uriPath, SolrParams params, ZkController
zkController) {
+
+// Validate that the node exists in the cluster
+if
(!zkController.zkStateReader.getClusterState().getLiveNodes().contains(nodeName))
{
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Requested node " + nodeName + " is not part of cluster");
Review Comment:
Agree that this may be a safer place to check that the node is actually part
of the cluster instead of earlier.
When moving from `nodes` to `node` parameter for prometheus, it was an
omission of me to not add a liveNodes check since we no longer call
`resolveNodes`.
The only downside I can see from this change is with other endpoints, e.g. a
call `/admin/info/system?nodes=foo:8983_solr,bad_node`, then earlier we'd fail
the request before proxying at all, while now we'll send the request to
`foo:8983`, while `bad_node` will throw exception and likely fail the entire
request eventually.
Being a corner case and no real harm, I think this is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on PR #3908: URL: https://github.com/apache/solr/pull/3908#issuecomment-3601624226 > > Starting with 10, can we at least have specifically the metrics be merely a single node that's proxied? IMO that's at least the right direction. And hopefully it'd spare the rather complex metrics processing here, and buffering. > > Not sure I understand this comment. But I **think** you may mean that the UI instead of requesting metrics from all nodes in one go with the `nodes=X,Y,Z` param, it could fire three requests, one with `nodes=X`, one `nodes=Y` and noe `nodes=Z`? Definitely a possibility. I implemented this. So now backend does no merging, but supports a single `&node=` param that will fetch metrics for that node. The frontend now loops over selected (visible) nodes in the cloud screen, and fetches metrics for each in parallel (from the same backend, but with different `node` param). This removes lots of code from backend, no more inserting "node" label in prom text, no more merging all lines in one response, and smaller tmp String object being cached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2580269555
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
+if (!prometheusText.isEmpty()) {
+ // Inject node label into each metric line
+ String labeledText = injectNodeLabelIntoText(prometheusText,
entry.getKey());
+ mergedText.append(labeledText);
Review Comment:
Added a comment line on top of response clarifying it is ONLY for UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908: URL: https://github.com/apache/solr/pull/3908#discussion_r2580270307 ## solr/webapp/web/js/angular/services.js: ## Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on PR #3908: URL: https://github.com/apache/solr/pull/3908#issuecomment-3600807871 > Starting with 10, can we at least have specifically the metrics be merely a single node that's proxied? IMO that's at least the right direction. And hopefully it'd spare the rather complex metrics processing here, and buffering. Not sure I understand this comment. But I **think** you may mean that the UI instead of requesting metrics from all nodes in one go with the `nodes=X,Y,Z` param, it could fire three requests, one with `nodes=X`, one `nodes=Y` and noe `nodes=Z`? Definitely a possibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908: URL: https://github.com/apache/solr/pull/3908#discussion_r2580136974 ## solr/webapp/web/js/angular/services.js: ## Review Comment: Ah, see it, will adapt plugins.js to read the new response from `Metrics` instead of a string wrapped in a json `data` key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2579848874
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
Review Comment:
Payload from each node is fairly limited unless you have 100 cores.. But def
room for further complexity 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2579836694
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
+if (!prometheusText.isEmpty()) {
+ // Inject node label into each metric line
+ String labeledText = injectNodeLabelIntoText(prometheusText,
entry.getKey());
+ mergedText.append(labeledText);
Review Comment:
Yea, it’s not supposed to be a new Prometheus exporter 😉 I suppose same
metrics would need to be grouped together under same comment headers for it to
be consumable by prom itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
Jesssullivan commented on PR #3908: URL: https://github.com/apache/solr/pull/3908#issuecomment-3598613161 > Starting with 10, can we at least have specifically the metrics be merely a single node that's proxied? IMO that's at least the right direction. And hopefully it'd spare the rather complex metrics processing here, and buffering. This makes sense to me, given it seems like the tilt for 10 is to default in Cloud Mode with deeper and more canonical instrumentation options via ZK 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
dsmiley commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2577654310
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
Review Comment:
sad... hopefully won't be too much? I hate buffering to complete response
size. I realize full streaming is harder. And fun to solve IMO :-)
BTW use
`org.apache.solr.client.solrj.response.InputStreamResponseParser#consumeResponseToString`
to save some code if we continue with reading them all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
mlbiscoc commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2577466685
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
+if (!prometheusText.isEmpty()) {
+ // Inject node label into each metric line
+ String labeledText = injectNodeLabelIntoText(prometheusText,
entry.getKey());
+ mergedText.append(labeledText);
Review Comment:
This merging works for the UI but this proxy response will most likely not
work for if someone intends to use this for their prometheus backend. I think
this is fine though and acceptable as it seems we never advertised this
previously and a user should never really be using this for prometheus scraping.
##
solr/webapp/web/js/angular/services.js:
##
Review Comment:
I think this change broke the `` -> `plugins / stats` page that I
changed to read prometheus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
Copilot commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2576163348
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +134,169 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+ZkController zkController = container.getZkController();
+Map>> responses = new LinkedHashMap<>();
+
+// Ensure wt=prometheus for all requests
+ModifiableSolrParams prometheusParams = new ModifiableSolrParams(params);
+if (!prometheusParams.get("wt", "").equals("prometheus")) {
+ prometheusParams.set("wt", "prometheus");
+}
+
+// Submit all requests
+for (String node : nodes) {
+ responses.put(node, callRemoteNode(node, pathStr, prometheusParams,
zkController));
+}
+
+// Collect all Prometheus text responses
+StringBuilder mergedText = new StringBuilder();
+for (Map.Entry>> entry :
responses.entrySet()) {
+ try {
+NamedList resp =
+entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+
+// Extract text from InputStream response
+Object streamObj = resp.get("stream");
+if (streamObj instanceof InputStream) {
+ try (InputStream stream = (InputStream) streamObj) {
+String prometheusText = new String(stream.readAllBytes(),
StandardCharsets.UTF_8);
+if (!prometheusText.isEmpty()) {
+ // Inject node label into each metric line
+ String labeledText = injectNodeLabelIntoText(prometheusText,
entry.getKey());
+ mergedText.append(labeledText);
+}
+ } catch (IOException ioe) {
+log.warn("IOException when reading stream from node {}",
entry.getKey(), ioe);
+ }
+} else {
+ log.warn("No stream in response from node {}", entry.getKey());
+}
+ } catch (ExecutionException ee) {
+log.warn("Exception when fetching Prometheus result from node {}",
entry.getKey(), ee);
+ } catch (TimeoutException te) {
+log.warn("Timeout when fetching Prometheus result from node {}",
entry.getKey(), te);
+ }
+}
+
+// Store the merged text in response - will be written as-is
+rsp.add("prometheusText", mergedText.toString());
+ }
+
+ /**
+ * Escape special characters in Prometheus label values according to
Prometheus specification.
+ * Escapes backslash, double quote, and newline characters.
+ */
+ private static String escapePrometheusLabelValue(String value) {
+return value.replace("\\", "").replace("\"", "\\\"").replace("\n",
"\\n");
+ }
+
+ /**
+ * Inject node="nodeName" label into Prometheus text format. Each metric
line gets the node label
+
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
janhoy commented on PR #3908: URL: https://github.com/apache/solr/pull/3908#issuecomment-3595290672 I'd like to merge this to unblock 10.0 if someone can give a thumbs up or a review comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
Copilot commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2573770351
##
solr/webapp/web/js/angular/controllers/cloud.js:
##
@@ -394,118 +394,134 @@ var nodesSubController = function($scope, Collections,
System, Metrics) {
Fetch metrics for all selected nodes. Only pull the metrics that we'll
show to save bandwidth
Pick the data we want to display and add it to the node-centric data
structure
*/
-Metrics.get({
- "nodes": nodesParam,
- "prefix":
"CONTAINER.fs,org.eclipse.jetty.server.handler.DefaultHandler.get-requests,INDEX.sizeInBytes,SEARCHER.searcher.numDocs,SEARCHER.searcher.deletedDocs,SEARCHER.searcher.warmupTime"
-},
-function (metricsResponse) {
- for (var node in metricsResponse) {
-if (node in nodes) {
- var m = metricsResponse[node];
- nodes[node]['metrics'] = m;
- var diskTotal =
m.metrics['solr.node']['CONTAINER.fs.totalSpace'];
- var diskFree =
m.metrics['solr.node']['CONTAINER.fs.usableSpace'];
- var diskPercentage = Math.floor((diskTotal - diskFree) /
diskTotal * 100);
- nodes[node]['diskUsedPct'] = diskPercentage;
- nodes[node]['diskUsedPctStyle'] = styleForPct(diskPercentage);
- nodes[node]['diskTotal'] = bytesToSize(diskTotal);
- nodes[node]['diskFree'] = bytesToSize(diskFree);
-
- var r =
m.metrics['solr.jetty']['org.eclipse.jetty.server.handler.DefaultHandler.get-requests'];
- nodes[node]['req'] = r.count;
- nodes[node]['req1minRate'] = Math.floor(r['1minRate'] * 100) /
100;
- nodes[node]['req5minRate'] = Math.floor(r['5minRate'] * 100) /
100;
- nodes[node]['req15minRate'] = Math.floor(r['15minRate'] * 100) /
100;
- nodes[node]['reqp75_ms'] = Math.floor(r['p75_ms']);
- nodes[node]['reqp95_ms'] = Math.floor(r['p95_ms']);
- nodes[node]['reqp99_ms'] = Math.floor(r['p99_ms']);
-
- // These are the cores we _expect_ to find on this node
according to the CLUSTERSTATUS
- var cores = nodes[node]['cores'];
- var indexSizeTotal = 0;
- var indexSizeMax = 0;
- var docsTotal = 0;
- var graphData = [];
- for (let coreId in cores) {
-var core = cores[coreId];
-if (core['shard_state'] !== 'active' || core['state'] !==
'active') {
- // If core state is not active, display the real state, or
if shard is inactive, display that
- var labelState = (core['state'] !== 'active') ?
core['state'] : core['shard_state'];
- core['label'] += "_(" + labelState + ")";
-}
-var coreMetricName = "solr.core." + core['collection'] + "." +
core['shard'] + "." + core['replica'];
-var coreMetric = m.metrics[coreMetricName];
-// we may not actually get metrics back for every expected
core (the core may be down)
-if (coreMetric) {
- var size = coreMetric['INDEX.sizeInBytes'];
- size = (typeof size !== 'undefined') ? size : 0;
- core['sizeInBytes'] = size;
- core['size'] = bytesToSize(size);
- indexSizeTotal = indexSizeTotal + size;
- indexSizeMax = size > indexSizeMax ? size : indexSizeMax;
- var numDocs = coreMetric['SEARCHER.searcher.numDocs'];
- numDocs = (typeof numDocs !== 'undefined') ? numDocs : 0;
- core['numDocs'] = numDocs;
- core['numDocsHuman'] = numDocsHuman(numDocs);
- core['avgSizePerDoc'] = bytesToSize(numDocs === 0 ? 0 : size
/ numDocs);
- var deletedDocs =
coreMetric['SEARCHER.searcher.deletedDocs'];
- deletedDocs = (typeof deletedDocs !== 'undefined') ?
deletedDocs : 0;
- core['deletedDocs'] = deletedDocs;
- core['deletedDocsHuman'] = numDocsHuman(deletedDocs);
- var warmupTime = coreMetric['SEARCHER.searcher.warmupTime'];
- warmupTime = (typeof warmupTime !== 'undefined') ?
warmupTime : 0;
- core['warmupTime'] = warmupTime;
- docsTotal += core['numDocs'];
-}
- }
- for (let coreId in cores) {
-var core = cores[coreId];
-var graphObj = {};
-graphObj['label'] = core['label'];
-graphObj['size'] = core['sizeInBytes'];
-graphObj['sizeHuman'] = core['size'];
-graphObj['pct'] = (core['sizeInBytes'] / indexSizeMax) * 100;
-graphData.push(graphObj);
- }
- if (cores) {
-cores.sort(function
Re: [PR] SOLR-18004 Admin UI nodes view to parse Prometheus metrics [solr]
Copilot commented on code in PR #3908:
URL: https://github.com/apache/solr/pull/3908#discussion_r2573656667
##
solr/core/src/java/org/apache/solr/handler/admin/AdminHandlersProxy.java:
##
@@ -129,9 +140,218 @@ public static CompletableFuture>
callRemoteNode(
URI baseUri =
URI.create(zkController.zkStateReader.getBaseUrlForNodeName(nodeName));
SolrRequest proxyReq = new GenericSolrRequest(SolrRequest.METHOD.GET,
uriPath, params);
+// Set response parser based on wt parameter to ensure correct format is
used
+String wt = params.get("wt");
+if ("prometheus".equals(wt) || "openmetrics".equals(wt)) {
+ proxyReq.setResponseParser(new InputStreamResponseParser(wt));
+}
+
return zkController
.getCoreContainer()
.getDefaultHttpSolrClient()
.requestWithBaseUrl(baseUri.toString(), c -> c.requestAsync(proxyReq));
}
+
+ /**
+ * Resolve node names from the "nodes" parameter into a set of live node
names.
+ *
+ * @param nodeNames the value of the "nodes" parameter ("all" or
comma-separated node names)
+ * @param container the CoreContainer
+ * @return set of resolved node names
+ * @throws SolrException if node format is invalid or node is not in cluster
+ */
+ private static Set resolveNodes(String nodeNames, CoreContainer
container) {
+Set liveNodes =
+
container.getZkController().zkStateReader.getClusterState().getLiveNodes();
+
+if (nodeNames.equals("all")) {
+ log.debug("All live nodes requested");
+ return liveNodes;
+}
+
+Set nodes = new HashSet<>(Arrays.asList(nodeNames.split(",")));
+for (String nodeName : nodes) {
+ if (!nodeName.matches("^[^/:]+:\\d+_[\\w/]+$")) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST, "Parameter " + PARAM_NODES +
" has wrong format");
+ }
+ if (!liveNodes.contains(nodeName)) {
+throw new SolrException(
+SolrException.ErrorCode.BAD_REQUEST,
+"Requested node " + nodeName + " is not part of cluster");
+ }
+}
+log.debug("Nodes requested: {}", nodes);
+return nodes;
+ }
+
+ /** Handle Prometheus format by fetching from nodes and merging text
responses. */
+ private static void handlePrometheusFormat(
+ Set nodes,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container,
+ SolrQueryResponse rsp)
+ throws IOException, SolrServerException, InterruptedException {
+
+// Bounded parallel executor - max concurrent fetches using Solr's
ExecutorUtil
+ExecutorService executor =
+new ExecutorUtil.MDCAwareThreadPoolExecutor(
+PROMETHEUS_PROXY_THREAD_POOL_SIZE, // corePoolSize
+PROMETHEUS_PROXY_THREAD_POOL_SIZE, // maximumPoolSize
+60L,
+TimeUnit.SECONDS,
+new LinkedBlockingQueue<>(),
+new SolrNamedThreadFactory("metricsProxyExecutor"));
+
+try {
+ // Submit all fetches at once - executor will handle bounded parallelism
+ Map> futures = new LinkedHashMap<>();
+ for (String node : nodes) {
+futures.put(node, fetchNodePrometheusTextAsync(executor, node,
pathStr, params, container));
+ }
+
+ // Collect all Prometheus text responses
+ StringBuilder mergedText = new StringBuilder();
+ for (Map.Entry> entry : futures.entrySet()) {
+try {
+ String prometheusText =
+ entry.getValue().get(PROMETHEUS_FETCH_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+ if (prometheusText != null && !prometheusText.isEmpty()) {
+// Inject node label into each metric line
+String labeledText = injectNodeLabelIntoText(prometheusText,
entry.getKey());
+mergedText.append(labeledText);
+ }
+} catch (ExecutionException ee) {
+ log.warn("Exception when fetching Prometheus result from node {}",
entry.getKey(), ee);
+} catch (TimeoutException te) {
+ log.warn("Timeout when fetching Prometheus result from node {}",
entry.getKey(), te);
+}
+ }
+
+ // Store the merged text in response - will be written as-is
+ rsp.add("prometheusText", mergedText.toString());
+
+} finally {
+ ExecutorUtil.shutdownAndAwaitTermination(executor);
+}
+ }
+
+ /** Fetch Prometheus text from a remote node asynchronously. */
+ private static Future fetchNodePrometheusTextAsync(
+ ExecutorService executor,
+ String nodeName,
+ String pathStr,
+ SolrParams params,
+ CoreContainer container) {
+
+return executor.submit(
+() -> {
+ try {
+ZkController zkController = container.getZkController();
+if (zkController == null) {
+ log.warn("ZkController not available for node {}", nodeName);
+ return null;
+}
+
+// Ensure wt=prometheus is set for inter-node requests
+Modifiab
