On Thursday, 19 March 2020 11:04:56 UTC, Jack Chew wrote: > > When i use node_exporter web-config.yml for TLS setting will arsie the > screenshot error. I try to different node_exporter server also same > promble, but i cancel web-config is no promblem. >
node_exporter 1.0.0 rc1? TLS with client cert authentication works for me. But if you're doing full TLS with client certs, you need authentication in both directions: - The server (node_exporter) needs a certificate signed by a CA - The name in the certificate (CN or SAN) needs to match either the hostname that prometheus is connecting to, or the "server_name" setting in tls_config <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config> if that is specified - The client (prometheus) needs a certificate signed by a CA [not necessarily the same one] - The server (node_exporter) doesn't care about the identity in the certificate, but it does need the CA cert of the certificate which signed prometheus' cert. Here's how I make this work with two keys and certs: one for prometheus, and one shared by all the node_exporters. I am going to assume you do the following on the prometheus server, and node_exporter is also running on this node (reachable as 127.0.0.1:9100), and show how to build it up in stages. 1. create a key and certificate for node_exporter to use: mkdir /etc/prometheus/ssl cd /etc/prometheus/ssl openssl req -x509 -newkey rsa:1024 -keyout prom_node_key.pem -out prom_node_cert.pem -days 29220 -nodes -subj /commonName=prom_node/ Type `ls` and you should see two files: `prom_node_cert.pem` and `prom_node_key.pem`. This is how the node_exporter identifies itself to prometheus. 2. create a file `/etc/prometheus/node_tls.yml` with the following contents: tlsConfig: tlsCertPath: /etc/prometheus/ssl/prom_node_cert.pem tlsKeyPath: /etc/prometheus/ssl/prom_node_key.pem 3. Change your node_exporter config to add --web.config=/etc/prometheus/node_tls.yml to the command-line options it runs with (e.g. edit your systemd unit file, or /etc/default/node_exporter, or whatever). Restart it and check for errors. 4. Now we can do a test scrape using curl and https: curl --cacert /etc/prometheus/ssl/prom_node_cert.pem --resolve prom_node:9100:127.0.0.1 -v https://prom_node:9100/metrics The scrape should be successful. We've done it over https. We've used the fake hostname "prom_node" to match the certificate, and told curl to use address 127.0.0.1 for this hostname, and to verify the certificate in prom_node_cert.pem. If it doesn't work at this point, fix the problem before proceeding. However, still anyone is authorized to scrape. So now we need to make a new key and cert for the prometheus server to use when scraping, and configure node_exporter so that it only accepts scrapes from someone with this key. 5. Create the new key and cert for prometheus: cd /etc/prometheus/ssl openssl req -x509 -newkey rsa:1024 -keyout prometheus_key.pem -out prometheus_cert.pem -days 29220 -nodes -subj /commonName=prometheus/ 6. Edit `/etc/prometheus/node_tls.yml` so it looks like this: tlsConfig: tlsCertPath: /etc/prometheus/ssl/prom_node_cert.pem tlsKeyPath: /etc/prometheus/ssl/prom_node_key.pem clientAuth: RequireAndVerifyClientCert clientCAs: /etc/prometheus/ssl/prometheus_cert.pem Restart node_exporter. 7. Now re-run the *exact* same curl command as you did before: curl --cacert /etc/prometheus/ssl/prom_node_cert.pem --resolve prom_node:9100:127.0.0.1 -v https://prom_node:9100/metrics This time you should see an error: curl: (35) gnutls_handshake() failed: Certificate is bad This is because the client isn't presenting a certificate to the server to identify itself. We now need to give a longer curl line (split for clarity): curl --cert /etc/prometheus/ssl/prometheus_cert.pem \ --key /etc/prometheus/ssl/prometheus_key.pem \ --cacert /etc/prometheus/ssl/prom_node_cert.pem \ --resolve prom_node:9100:127.0.0.1 \ -v https://prom_node:9100/metrics This should now work. We've proved our identity to node_exporter using the prometheus private key, and node_exporter will now talk to us. 8. Now you just need to change the prometheus config to scrape using tls. Edit your prometheus.yml and find the section which scrapes node_exporter. Edit it so that it includes scheme: https and a tls_config section as below. - job_name: 'node' file_sd_configs: - files: - /etc/prometheus/targets.d/node.yml scheme: https tls_config: # Verifying remote identity ca_file: /etc/prometheus/ssl/prom_node_cert.pem server_name: prom_node # Asserting our identity cert_file: /etc/prometheus/ssl/prometheus_cert.pem key_file: /etc/prometheus/ssl/prometheus_key.pem Signal prometheus to re-read its configuration, and check for errors: killall -HUP prometheus journalctl -eu prometheus # e.g. if you are running prometheus under systemd 9. Deployment to other nodes To deploy this to remote nodes with node_exporter, you would copy the following files to them: * `/etc/default/node_exporter` (or however you set the command line options on node_exporter) * `/etc/prometheus/node_tls.yml` * `/etc/prometheus/ssl/prom_node_cert.pem` * `/etc/prometheus/ssl/prom_node_key.pem` * `/etc/prometheus/ssl/prometheus_cert.pem` but *NOT* `prometheus_key.pem`. That file is private to the prometheus server only; it's ownership of this key which proves the prometheus server's identity. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7ef89b3f-0ccd-4f6b-9988-db029e0f8855%40googlegroups.com.
