Re: [tor-relays] tor prometheus metrics -> expected label value, got "INVALID"

2022-01-24 Thread Fran via tor-relays 

Hej David,


Thanks for reporting and investigating!! :)


Thank you for fixing and implementing in the first place!

Really looking forward to see some nice graphs showing how many
people use our services via onion.

BTW: Implementation of nusenu feature request[1] for exposing more relay 
health metrics for prometheus would be awesome! :)


Best,
fran

[1] https://gitlab.torproject.org/tpo/core/tor/-/issues/40194
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] tor prometheus metrics -> expected label value, got "INVALID"

2022-01-24 Thread David Goulet 
On 23 Jan (11:36:05), Fran via tor-relays wrote:
> Hej,
> 
> taking another look and comparing it to onion_services exporter there is a
> slight difference in the metrics output.
> 
> prometheus-onion-service-exporter:
> 
> onion_service_up{address="foobar:587",name="mail_v3_587",type="tcp"} 1
> 
> vs output from tor:
> 
> tor_hs_app_write_bytes_total{onion=foobar,port=80} 298891

Ah wow... you are indeed right.

I had problems recently with an exporter I made for something unrelated that
was not exporting labels as quoted strings...

I have opened a ticket:

https://gitlab.torproject.org/tpo/core/tor/-/issues/40552

Thanks for reporting and investigating!! :)

Cheers!
David

-- 
3obTrBY9FLhQsK4/YufSTiyO7ERRvmLFjMaVvocpZfQ=


signature.asc
Description: PGP signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

Re: [tor-relays] tor prometheus metrics -> expected label value, got "INVALID"

2022-01-23 Thread Fran via tor-relays 

Hej,

taking another look and comparing it to onion_services exporter there is 
a slight difference in the metrics output.


prometheus-onion-service-exporter:

onion_service_up{address="foobar:587",name="mail_v3_587",type="tcp"} 1

vs output from tor:

tor_hs_app_write_bytes_total{onion=foobar,port=80} 298891

The working onion-service-exporter double quotes the labels but tor doesn't.

Debug messages from prometheus are also not very helpful:

[...] msg="Append failed" err="expected label value, got \"INVALID\""

But after searching around for a long time I found that promtool can 
help with debugging scraping erros.


Promtool complains about the missing double quotes

$ GET http://:9035/metrics | promtool check metrics
error while linting: text format parsing error in line 3: expected '"' 
at start of label value, found 'f'


Saving the exporter output to a file and quoting the onion and port 
label values:


$ cat quoted-output.txt | promtool check metrics
tor_hs_intro_established_count non-histogram and non-summary metrics 
should not have "_count" suffix
tor_hs_rdv_established_count non-histogram and non-summary metrics 
should not have "_count" suffix


After removing the criticized "_count" in the file promtool is happy.

So, how to fix this to not make my weekend project fail? Relabel label 
values!


First try (for testing):

relabel_configs:
- action: labeldrop
  regex: "port"
- action: labeldrop
  regex: "onion"

not working, still 'expected label value, got "INVALID"'. What about:

relabel_configs:
- target_label: onion
  replacement: foo
- target_label: port
  replacement: bar

now I got not only the labels 'instance=":9035"job="tor_metrics"' 
but also 'onion="foo" port="bar"' but still the 'expected label value, 
got "INVALID"' error.


Out of ideas for the moment.

Best fran


On 1/23/22 01:20, Fran wrote:

Hej,

I just found out about the integrated prometheus exporter in tor - yay, 
thanks for that!


I can scrape it with curl:

$ curl http://:9035/metrics
# HELP tor_hs_rdv_num_total Total number of rendezvous circuit created
# TYPE tor_hs_rdv_num_total counter
tor_hs_rdv_num_total{onion=} 4
# HELP tor_hs_app_write_bytes_total Total number of bytes written to the 
application

# TYPE tor_hs_app_write_bytes_total counter
tor_hs_app_write_bytes_total{onion=,port=80} 15211
# HELP tor_hs_intro_num_total Total number of introduction received
# TYPE tor_hs_intro_num_total counter
tor_hs_intro_num_total{onion=} 4
# HELP tor_hs_rdv_established_count Total number of established 
rendezvous circuit

# TYPE tor_hs_rdv_established_count gauge
tor_hs_rdv_established_count{onion=} 1
# HELP tor_hs_app_read_bytes_total Total number of bytes read from the 
application

# TYPE tor_hs_app_read_bytes_total counter
tor_hs_app_read_bytes_total{onion=,port=80} 319667
# HELP tor_hs_intro_established_count Total number of established 
introduction circuit

# TYPE tor_hs_intro_established_count gauge
tor_hs_intro_established_count{onion=} 18

Which looks like proper data for me. Unfortunately prometheus thinks 
otherwise. In the prometheus webGUI under "/targets" it complains:


 > expected label value, got "INVALID"

The Prometheus config part is:

   - job_name: tor_metrics
     metrics_path: /metrics
     static_configs:
     - targets:
   - :9035
   - :9035

Servers are running Debian 11, Prometheus 2.32.1 and tor 
0.4.6.9-1~d11.bullseye+1



I searched the relays mailinglist and tried to find something in the 
internet, unfortunately not very successfully.


Any ideas?

Thanks a lot!

fran

___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

[tor-relays] tor prometheus metrics -> expected label value, got "INVALID"

2022-01-23 Thread Fran via tor-relays 

Hej,

I just found out about the integrated prometheus exporter in tor - yay, 
thanks for that!


I can scrape it with curl:

$ curl http://:9035/metrics
# HELP tor_hs_rdv_num_total Total number of rendezvous circuit created
# TYPE tor_hs_rdv_num_total counter
tor_hs_rdv_num_total{onion=} 4
# HELP tor_hs_app_write_bytes_total Total number of bytes written to the 
application

# TYPE tor_hs_app_write_bytes_total counter
tor_hs_app_write_bytes_total{onion=,port=80} 15211
# HELP tor_hs_intro_num_total Total number of introduction received
# TYPE tor_hs_intro_num_total counter
tor_hs_intro_num_total{onion=} 4
# HELP tor_hs_rdv_established_count Total number of established 
rendezvous circuit

# TYPE tor_hs_rdv_established_count gauge
tor_hs_rdv_established_count{onion=} 1
# HELP tor_hs_app_read_bytes_total Total number of bytes read from the 
application

# TYPE tor_hs_app_read_bytes_total counter
tor_hs_app_read_bytes_total{onion=,port=80} 319667
# HELP tor_hs_intro_established_count Total number of established 
introduction circuit

# TYPE tor_hs_intro_established_count gauge
tor_hs_intro_established_count{onion=} 18

Which looks like proper data for me. Unfortunately prometheus thinks 
otherwise. In the prometheus webGUI under "/targets" it complains:


> expected label value, got "INVALID"

The Prometheus config part is:

  - job_name: tor_metrics
metrics_path: /metrics
static_configs:
- targets:
  - :9035
  - :9035

Servers are running Debian 11, Prometheus 2.32.1 and tor 
0.4.6.9-1~d11.bullseye+1



I searched the relays mailinglist and tried to find something in the 
internet, unfortunately not very successfully.


Any ideas?

Thanks a lot!

fran
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays