Re: [tor-dev] onionoo.tpo stuck at 2018-01-21 22:00
On 2018-01-26 11:34, nusenu wrote: >> What we can do, though, is think about providing more history in >> Onionoo, so that you can give up on archiving Onionoo data. > > It is nice of you to consider that but it is not necessary (at least for me) > I can life with my current hacks and other probably don't need more history, > and you have already enough stuff on your plate. Okay. >>> Do you monitor onionoo for such problems ("relays_published" timestamp >>> remaining unchanged for >1-2 hours)? >>> Would you find something like that useful? >> >> We do have such monitoring, yes. > > So my email was redundant to your nagios check? > > Would it be possible to publish these alerts on a mailing list? :) Not a crazy idea! I opened a ticket for further discussing this: https://trac.torproject.org/projects/tor/ticket/25035 All the best, Karsten signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] onionoo.tpo stuck at 2018-01-21 22:00
> What we can do, though, is think about providing more history in > Onionoo, so that you can give up on archiving Onionoo data. It is nice of you to consider that but it is not necessary (at least for me) I can life with my current hacks and other probably don't need more history, and you have already enough stuff on your plate. >> Do you monitor onionoo for such problems ("relays_published" timestamp >> remaining unchanged for >1-2 hours)? >> Would you find something like that useful? > > We do have such monitoring, yes. So my email was redundant to your nagios check? Would it be possible to publish these alerts on a mailing list? :) -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] onionoo.tpo stuck at 2018-01-21 22:00
Hi nusenu, On 2018-01-22 18:57, nusenu wrote: >> Looks like the primary CollecTor instance had a problem between 22:00 >> and 08:00 UTC. It works again now, as does Onionoo. > > Karsten, thanks for the fast reaction. > >> We didn't lose any data, because the primary CollecTor instance obtained >> all descriptors it had missed earlier from the backup CollecTor instance. > > > Since I'm archiving onionoo data I'm "loosing" data (causing blind spots) > everytime a "relays_published" > timestamp is skipped. In theory one could spin up an onionoo instance to > generate data for skipped > timestamps but in practice this is hard (requires lots of resources). > (I know, you are probably talking about not loosing any raw CollecTor data, > but wanted to mention that > nonetheless.) Right, I meant not losing any raw CollecTor data. Your use case of archiving Onionoo data is special. It's okay that you do this, but it's not what Onionoo was designed for. Most people will find Onionoo data that is 6 or 12 hours behind still useful. But if we had lost 6 or 12 hours of CollecTor data, that would have been pretty bad. What we can do, though, is think about providing more history in Onionoo, so that you can give up on archiving Onionoo data. After all, Onionoo already provides quite some history, including graph data like in bandwidth documents and others, times when a relay last changed its IP address or port, the time it was first seen, and so on. If you have ideas what else would be valuable to have history for, please open a ticket. > Do you monitor onionoo for such problems ("relays_published" timestamp > remaining unchanged for >1-2 hours)? > Would you find something like that useful? We do have such monitoring, yes. Here's the Nagios script we're using: https://gitweb.torproject.org/admin/tor-nagios.git/tree/tor-nagios-checks/checks/tor-check-onionoo > Thanks for keeping it running besides all the other things you do. > > I'm wondering if the admin team would be available to cover such cases to > reduce > the operations load for developers. The admin team already handles operational issues with the hosts, though the metrics team is still in charge for running the services. I think that's a fine separation, and it has worked quite well for the last couple of years. > kind regards, > nusenu All the best, Karsten signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] onionoo.tpo stuck at 2018-01-21 22:00
> Looks like the primary CollecTor instance had a problem between 22:00 > and 08:00 UTC. It works again now, as does Onionoo. Karsten, thanks for the fast reaction. > We didn't lose any data, because the primary CollecTor instance obtained > all descriptors it had missed earlier from the backup CollecTor instance. Since I'm archiving onionoo data I'm "loosing" data (causing blind spots) everytime a "relays_published" timestamp is skipped. In theory one could spin up an onionoo instance to generate data for skipped timestamps but in practice this is hard (requires lots of resources). (I know, you are probably talking about not loosing any raw CollecTor data, but wanted to mention that nonetheless.) Do you monitor onionoo for such problems ("relays_published" timestamp remaining unchanged for >1-2 hours)? Would you find something like that useful? Thanks for keeping it running besides all the other things you do. I'm wondering if the admin team would be available to cover such cases to reduce the operations load for developers. kind regards, nusenu -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] onionoo.tpo stuck at 2018-01-21 22:00
On 2018-01-22 09:03, nusenu wrote: > Hi Karsten, Hi nusenu, > just wanted to let you know that the delta between > relays_published and current time is unusually high. > > > https://onionoo.torproject.org/details?limit=0 > > {"version":"5.0", > "build_revision":"0bce98a", > "relays_published":"2018-01-21 22:00:00", > > This is currently blocking ornetradar reports. Looks like the primary CollecTor instance had a problem between 22:00 and 08:00 UTC. It works again now, as does Onionoo. We didn't lose any data, because the primary CollecTor instance obtained all descriptors it had missed earlier from the backup CollecTor instance. > thanks for having a look, Thanks for the report! > nusenu All the best, Karsten signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev