plumdog commented on issue #272:
URL: https://github.com/apache/solr-operator/issues/272#issuecomment-1103138897
I now think this change is not actually needed to be able to gather JMX
metrics. I happened to read the following while doing something completely
different, from `kubectl explain.pod.spec.containers.ports`:
> Exposing a port here gives the system additional information about the
network connections a container uses, but is primarily informational. Not
specifying a port here DOES NOT prevent that port from being exposed. Any port
which is listening on the default "0.0.0.0" address inside a container will be
accessible from the network.
So I tried just setting some JMX options and pointing Datadog at the port.
My changes were (broadly) as follows:
In the values passed to the Datadog chart:
```yaml
agents:
image:
tagSuffix: jmx
```
And then in my `SolrCloud`:
```yaml
apiVersion: solr.bloomberg.com/v1beta1
kind: SolrCloud
# ...
spec:
customSolrKubeOptions:
podOptions:
annotations:
ad.datadoghq.com/solrcloud-node.check_names: '["solr"]'
ad.datadoghq.com/solrcloud-node.init_configs: '{"is_jmx": true,
"collect_default_metrics": true}'
ad.datadoghq.com/solrcloud-node.instances: '{"host": "%%host%%",
"port": 18983}'
solrOpts: >-
# ...
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=18983
-Dcom.sun.management.jmxremote.rmi.port=18983
```
And then I started getting the metrics from
https://github.com/DataDog/integrations-core/blob/master/solr/metadata.csv
arriving in Datadog (some took a while to come through, not sure if that's
because Datadog or because Solr).
For reference, I took the Solr options from
https://solr.apache.org/guide/8_9/using-jmx-with-solr.html#configuring-a-remote-connection-to-solr-jmx
and the Datadog options from
https://docs.datadoghq.com/agent/guide/autodiscovery-with-jmx/?tab=containeragent#autodiscovery-annotations
Some things my config above doesn't do that might be a good idea:
- use authentication when connecting to JMX from Datadog. In my case this
isn't really a concern, but I think is easy, so I might as well. I can imagine
situations where the JMX options I'm setting above would be bad.
- set the pod name as a tag for Datadog, which seems to be missing, so I'm
just getting `instance:solr-[podIP]-18983` which isn't lovely
- I've not looked at getting anything from Zookeeper.
(The above is quite specific to Datadog, but I think shows that this issue
doesn't block gathering of JMX metrics from pods. I hope it is a useful
reference.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
