Re: [tor-relays] Monitoring multiple relays
Vasilis: > Hi, > Reopening thread after IRC discussion. Bottom-posted, instead of more sensibly posting inline. > DaKnOb: >> It depends on what you consider “professional” monitoring. Do you >> mean information collected, or how was it collected? > > By professional monitoring I mean a way to find out in a short > time-span what was the reason for a relay that suddenly is > disconnected from the Tor network, uses an outdated version of tor, > performs badly on the Tor network, runs an outdated OS version, > misses security updates or other crucial software that may compromise > the Tor relays and subsequently the Tor network. > > Some important properties of this monitoring system: > > - Hardware issues: RAID/HD/hardware failures, kernel panic/OOM > states - Software issues: OS updates, tor updates, security updates - > Network issues: RBLs, IP blocking, upstream network issues - Abuse > issues: Monitor of abuse emails per relay/network -sort of ticketing > system for operators that are unwilling/don't know/have the capacity > to track and respond to abuse emails (that most of the time are > automated and just a 'foo' response back) - Legal issues: Initiating > a canary-like or similar for relay operators that would like to be > reached out when they don't provide any updates. I suspect this to > have many false positives but better safe that sorry (quite often you > are not allowed to speak openly about a legal issue until this is > settled, in this part potential organizations may reach out to help > operators) > >> Is measuring something from the tor process using bash scripts and >> cron professional? Is measuring network traffic using Prometheus >> and plotting to Grafana professional? > > My "professional point of view" will be a system -preferably > agent-less- that could ping operators via email and provide alert > notifications on an IRC channel. > >> For a few nodes I control / controlled I measured lots of network >> info such as: >> >> - Network Traffic in / out (b/s) - Network Packets in / out (p/s) - >> Network Flows in / out (f/s) >> >> And I always run a local resolver, so DNS info too: >> >> - Query Responses / Second - Query Latency - SERVFAILs / Second >> >> The DNS info was gathered only in one node, as an experiment, since >> I wasn’t sure whether it could leak information, and only for a >> limited amount of time. > > I share the same concerns with you so I'm not really interested in > measuring DNS responses or collecting long-term stats that may leak > sensitive information or potentially used to de-anomymize or > compromise in any way (in ways that we don't know yet) the Tor > network. From a quick read through on this, it seems there is a case for different tools instead of "one size fits all". I would divide the functions into three categories and therefore three tools: 1. remote checks of server responsiveness Most any tool could work here. I'm a long-time fan of sysmon (https://puck.nether.net/sysmon/). It's light, configurable, very modular with clean syntax and provides a good array of checks with email alerts. Most importantly for the stated purposes, no need for a local agent running on the target systems. 2. local system checks The BSDs do daily/weekly/monthly emails by default, with raid health checks and other tasks available or easily extendable with a little shell scripting. Or, I think opting for some shell scripts for checking the raid health, etc, would be fine. This doesn't scale well, obviously, when you're talking about a daily per system. But in that case, the shell script option or config management should work. Think about what you want, ie, "is the raid array dead", and go for it. 3. remote checks of Tor-related statistics How Tor is operating can be done one of two ways, I'm thinking off the top of my head. If you want periodic checks about consensus weight, or anything available through Onionoo from https://metrics.torproject.org/onionoo.html with JSON might make sense worked into some email output. g -- 34A6 0A1F F8EF B465 866F F0C5 5D92 1FD1 ECF6 1682 signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
Hi, DaKnOb: > It depends on what you consider “professional” monitoring. Do you mean > information collected, or how was it collected? By professional monitoring I mean a way to find out in a short time-span what was the reason for a relay that suddenly is disconnected from the Tor network, uses an outdated version of tor, performs badly on the Tor network, runs an outdated OS version, misses security updates or other crucial software that may compromise the Tor relays and subsequently the Tor network. Some important properties of this monitoring system: - Hardware issues: RAID/HD/hardware failures, kernel panic/OOM states - Software issues: OS updates, tor updates, security updates - Network issues: RBLs, IP blocking, upstream network issues - Abuse issues: Monitor of abuse emails per relay/network -sort of ticketing system for operators that are unwilling/don't know/have the capacity to track and respond to abuse emails (that most of the time are automated and just a 'foo' response back) - Legal issues: Initiating a canary-like or similar for relay operators that would like to be reached out when they don't provide any updates. I suspect this to have many false positives but better safe that sorry (quite often you are not allowed to speak openly about a legal issue until this is settled, in this part potential organizations may reach out to help operators) > Is measuring something from the tor process using bash scripts and cron > professional? > Is measuring network traffic using Prometheus and plotting to Grafana > professional? My "professional point of view" will be a system -preferably agent-less- that could ping operators via email and provide alert notifications on an IRC channel. > For a few nodes I control / controlled I measured lots of network info such > as: > > - Network Traffic in / out (b/s) > - Network Packets in / out (p/s) > - Network Flows in / out (f/s) > > And I always run a local resolver, so DNS info too: > > - Query Responses / Second > - Query Latency > - SERVFAILs / Second > > The DNS info was gathered only in one node, as an experiment, since I wasn’t > sure whether it could leak information, and only for a limited amount of > time. I share the same concerns with you so I'm not really interested in measuring DNS responses or collecting long-term stats that may leak sensitive information or potentially used to de-anomymize or compromise in any way (in ways that we don't know yet) the Tor network. ~Vasilis -- Fingerprint: 8FD5 CF5F 39FC 03EB B382 7470 5FBF 70B1 D126 0162 Pubkey: https://pgp.mit.edu/pks/lookup?op=get&search=0x5FBF70B1D1260162 signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
If you didn't do that already you should certainly speak with the tor metrics team since a new implementation of something like tor weather is on their roadmap (in ~12months). -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
It depends on what you consider “professional” monitoring. Do you mean information collected, or how was it collected? Is measuring something from the tor process using bash scripts and cron professional? Is measuring network traffic using Prometheus and plotting to Grafana professional? For a few nodes I control / controlled I measured lots of network info such as: - Network Traffic in / out (b/s) - Network Packets in / out (p/s) - Network Flows in / out (f/s) And I always run a local resolver, so DNS info too: - Query Responses / Second - Query Latency - SERVFAILs / Second The DNS info was gathered only in one node, as an experiment, since I wasn’t sure whether it could leak information, and only for a limited amount of time. Would be interested however to see what you’re building! Antonis > On 20 Oct 2017, at 19:34, Vasilis wrote: > > Hello, > > I'm reopening this thread as I would like to do some "professional" > monitoring on my relays and working on a solution that could be helpful > to other relay operators running few relays or don't want to go into the > hassle of deploying a monitoring system. > > My idea is to deploy a monitoring system that can support monitoring not > only for my relays and friends relays (people who trust me) but also to > other relays that have no time or resources to monitor their relays. > > Any suggestions, thoughts, comments and especially a 'I/we did X and > succeed/failed' are greatly appreciated. > > > Cheers, > ~Vasilis > -- > Fingerprint: 8FD5 CF5F 39FC 03EB B382 7470 5FBF 70B1 D126 0162 > Pubkey: https://pgp.mit.edu/pks/lookup?op=get&search=0x5FBF70B1D1260162 > > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
Hello, I'm reopening this thread as I would like to do some "professional" monitoring on my relays and working on a solution that could be helpful to other relay operators running few relays or don't want to go into the hassle of deploying a monitoring system. My idea is to deploy a monitoring system that can support monitoring not only for my relays and friends relays (people who trust me) but also to other relays that have no time or resources to monitor their relays. Any suggestions, thoughts, comments and especially a 'I/we did X and succeed/failed' are greatly appreciated. Cheers, ~Vasilis -- Fingerprint: 8FD5 CF5F 39FC 03EB B382 7470 5FBF 70B1 D126 0162 Pubkey: https://pgp.mit.edu/pks/lookup?op=get&search=0x5FBF70B1D1260162 signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
Hi, if you have a few relays I would go "the professional way" and set up an Icinga 2 server. If you combine Icinga 2 with a graphing tool like Graphite you can also produce nice graphs. You can monitor the whole server (CPU, RAM, disk space, Tor bandwidth, etc) and get alarmed when something is misbehaveing. ~Josef Am 24.05.2016 um 19:04 schrieb Xza: > Hello, > > I have a couple relays / exits running. > Now my question is : how do you manage them is there any dashboard or > CLI tools to manage them ( statistics, ect.. ) > I know the cli tool specially for Tor "arm" > Thanks alot. > > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
Exact, it can be a useful tool. All the servers you own can be shown in a list, and several tools to manage them like some VPS management where you got everything on the same place. I see some operators launching several Tor instances to use many cpu cores, so it can be nice to have something like this on the same server. For people who know murmurd (Mumble server, voip), there's a tool to manage your server(s), it's easy to set up a new instance for example... To have a look http://yulli.cleanvoice.ru/ I think this tool can be an example for Tor! But Mumble is listening on a special port to accept this kind of tool, Tor too if I'm not wrong... Sadly, I'm not a dev ! If it can give a nice idea to someone ;) Le 24/05/2016 19:04, Xza a écrit : > Hello, > > I have a couple relays / exits running. > Now my question is : how do you manage them is there any dashboard or CLI > tools to manage them ( statistics, ect.. ) > I know the cli tool specially for Tor "arm" > Thanks alot. > > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays > -- Petrusko PubKey EBE23AE5 C0BF 2184 4A77 4A18 90E9 F72C B3CA E665 EBE2 3AE5 signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Monitoring multiple relays
I use a third-party monitoring service to monitor the Dir and OR ports of all my relays. It's especially useful now that Tor Weather isn't maintained. The service constantly checks for a response from both ports, using several monitoring endpoints around the world, and notifies me of any downtime. It also has a nice feature where I can look for a specific response from the Dir port (i.e, parse http://relay1.example.com/tor/server/authority and look for the fingerprint). https://www.statuscake.com/ (the free tier is sufficient for this use case) I don't know of an easy way to aggregate the Tor stats in one place, but over time I've realized those stats don't matter too much anyway. :) As long as the nodes are running well and providing service I don't really need to know how many circuits or the bandwidth at any given moment. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Monitoring multiple relays
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hello, I have a couple relays / exits running. Now my question is : how do you manage them is there any dashboard or CLI tools to manage them ( statistics, ect.. ) I know the cli tool specially for Tor "arm" Thanks alot. - -- PGP : 29A4CE52 -BEGIN PGP SIGNATURE- iQI4BAEBCgAiGxxEbyA8eWFuZGVyZXNvbkByaXNldXAubmV0PgUCV0SJlwAKCRBI of/XNyszSvv/D/9VtjI+bmiqK53v//StUbmpYjgQjYheYvwktFld+2XDURh52dmU sbk3lab2RS1eQJxjV0BVXMF9O2JUAlp4NA2UMpj8ZJlCk0h1S6p2o4/sAlsrHr8Z nb67HiaRIyEvi9cQiXFtDtK/onAkEuKskolyvh3ysJRRUubHSX1z8WuNJIc4xyGH ZpSLPGUOO6qI7mM8ztS0muEy1khmkHgtlyHi2l4K/NsW3I90cV3w6Db2AEnV6v1c tsfehqLExsHUdEo0U3hHU7yY1p2fzD5Z+C0tgVKTnP5Zus/Vr9OH/zFwxneo0E7a 5FHdnCZnNIG9W32uQgjDdmBEv/YFEd0um/vMWFGUehFANSt+DIps7E+AjwiJwm2i ov97JJurQkrSMX7ZchTEsy720Ju272olp2eDiHaXXTK2TkxcEq+d1gZfj7CXycPW 6mCrj6L6URJAZ34GJQfFFjv8QbFmKkAvGhxl/O8KXRbXxedbLH7PY73jmFjJcmxO +KnSeyqMe1KbzazYyFydP+VmgI2k1IlO89l8Jq1Gq9f2KOPY9UKFZjGDKLS0xGZW EnqU/KUINcrGejYFT8nYNxA+OFTd/ZGIX5qP4Vo5NsZ/pDLZjdxIcvHx1jjZbER+ 5Ku2bihm2KTNAGoYDqB3LGCm+QdEFG/p573xQCcXPk92GyhMJDi9EMPDgA== =mTa0 -END PGP SIGNATURE- ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
