Re: [tor-relays] A new kind of attack?
Hi > > I've noticed a new kind of possible attack on some of my relays, as > > early as Dec.23 which causes huge spikes of outbound traffic that > > eventually maxes out RAM and crashes Tor. The newest one today > > lasted for 5 hours switching between two of the three relays on the > > same IP. > > > > I have included charts and excerpts from the log in my post in Tor > > forum at below link: > > > > https://forum.torproject.org/t/new-kind-of-attack/11122 > > I've noticed this as well, on 0.4.8.10 across FreeBSD and Alpine > platforms, against relays too new to receive any meaningful traffic > from regular clients. MaxMemInQueues does not prevent the relay's > eventual saturation of available memory on the system. The relays > operated as exit nodes. > > We're low on memory (cell queues total alloc: 6336 buffer total > alloc: 1556480, tor compress total alloc: 1073827425 (zlib: 0, zstd: > 0, lzma: 1073827249), rendezvous cache total alloc: 0). Killing > circuits│withover-long queues. (This behavior is controlled by > MaxMemInQueues.) I attached what is a typical picture for my entry relays. Between normal and aggressive is a factor of 3-20 in circuits. The relay frontside (inbound) seems not impacted. Tor 0.4.8.9 running on FreeBSD with Libevent 2.1.12-stable, OpenSSL LibreSSL 3.7.3, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.5 and BSD 1302001 as libc. MaxMemInQueues 2 GB 2023-12-31, normal The relay takes 3216M memory and 9k files are open MM DD hh mm Circuits txGB rxGB ConnIp4rx ConnIp6rx ConnIp4tx ConnIp6tx 2023 12 31 00 55 41386 24 23 9165 563 2834 381 2023 12 31 01 55 39220 22 22 8550 472 2517 356 2023 12 31 02 55 38644 22 22 8411 456 2312 324 2023 12 31 03 55 40609 21 20 8650 496 2623 466 2023 12 31 04 55 37846 22 21 8424 504 3078 519 2023 12 31 05 55 35218 21 22 8210 457 2872 513 2023 12 31 06 55 35851 24 23 8126 472 2748 430 2023 12 31 07 55 35074 24 23 7971 404 2502 335 2023 12 31 08 55 34321 22 23 8069 448 2332 309 2023 12 31 09 55 35283 21 19 7909 430 1913 283 2023 12 31 10 55 33941 21 21 7709 457 1790 285 2023 12 31 11 55 33825 20 20 7643 484 1884 276 2023 12 31 12 55 32752 24 23 7328 474 1877 196 2023 12 31 13 55 32823 21 21 7333 511 1843 227 2023 12 31 14 55 29976 28 28 7058 473 1680 244 2023 12 31 15 55 28559 25 24 7096 503 1701 292 2023 12 31 16 55 28873 24 24 7217 493 1722 440 2023 12 31 17 55 29011 19 19 6994 487 1674 456 2023 12 31 18 55 32967 21 20 6710 455 1554 363 2023 12 31 19 55 28556 21 21 6714 450 1466 262 2023 12 31 20 55 27904 21 21 6558 384 1507 247 2023 12 31 21 55 27409 22 22 6130 390 1505 231 2023 12 31 22 55 26879 23 22 5929 390 1458 219 2023 12 31 23 55 25827 22 22 5627 376 1333 218 2024 01 01 00 55 28670 17 17 5955 451 1324 276 2024-01-11, aggressive The relay takes 7502M memory and 10k files are open MM DD hh mm Circuits txGB rxGB ConnIp4rx ConnIp6rx ConnIp4tx ConnIp6tx 2024 01 11 00 55 125285 30 30 12105 900 3399 648 2024 01 11 01 55 110064 30 29 11827 995 3725 790 2024 01 11 02 55 45423 24 22 13148 633 2549 543 2024 01 11 03 55 99047 21 20 12944 710 2363 444 2024 01 11 04 55 122485 23 22 11627 705 3623 543 2024 01 11 05 55 113557 23 23 9414 701 3842 709 2024 01 11 06 55 115456 23 23 9265 760 3980 934 2024 01 11 07 55 114597 22 22 9428 798 3733 904 2024 01 11 08 55 120269 27 27 10494 824 3652 702 2024 01 11 09 55 117867 27 25 9936 822 3774 740 2024 01 11 10 55 115923 31 31 9441 812 3734 752 2024 01 11 11 55 116081 28 28 9861 852 3850 714 2024 01 11 12 55 109707 25 24 10266 913 3639 659 2024 01 11 13 55 340445 48 29 15059 1750 3565 623 2024 01 11 14 55 637652 100 16 15583 1594 3886 824 2024 01 11 15 55 553291 100 13 10128 790 3410 700 2024 01 11 16 55 599953 97 16 19689 2965 3293 625 2024 01 11 17 55 559004 100 20 19513 3108 2743 545 2024 01 11 18 55 854193 90 18 51 664 3908 580 2024 01 11 19 55 752697 84 16 13 643 4069 749 2024 01 11 20 55 65342 47 8 17236 2092 2663 663 2024 01 11 21 55 42592 5 4 7842 334 2502 562 2024 01 11 22 55 118705 17 15 11781 781 4688 1169 2024 01 11 23 55 129431 23 23 12623 1145 4946 1128 2024 01 12 00 55 123173 22 21 13507 1154 4759 1119 -- Cheers, Felix pgpZsBCLxI6x5.pgp Description: Digitale Signatur von OpenPGP ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Recent Tor versions not reloading config on / ignoring HUP kill signal.
On 1/13/24 18:29, George Hartley via tor-relays wrote: Is anyone else experiencing this? Yes, https://gitlab.torproject.org/tpo/core/tor/-/issues/40749 -- Toralf ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay not connecting
Hi >Sorry, but I'm going to vent a little bit. I'm not a network >specialist, so 11 days ago I sent the following email to this > mailing list asking for help because two of my Tor exit relays were > completely frozen and I was unable to put them online again. According to https://metrics.torproject.org/rs.html#details/3B85067588C3F017D5CCF7D8F65B5881B7D4C97C the relay is back since 1-2 days, good. Exiting to port 22 might lead to a lot of complaints ending at your ISP or yourself. Default SSH. >Nobody answered, not even a comment. No wait, there was one person Unfortunately that happens from time to time. Thanks for your good report. Did you check https://gitlab.torproject.org/tpo/core/tor/-/issues/ for the bug you reported? > - very active on this mailing list recently - who sent me an email >personally to tell me that I was an idiot (referring to what, I > don't know) who should kill himself. Adding furthermore that if I > didn't commit suicide within 72 hours, he would personally come to my > house and kill me with a Glock 9 mm. Fun stuff, very disturbing. Nobody should read or write something like that. Makes me sad. >Anyway, no serious answers, someone calling me an idiot: I tried to >look as best as I could at what I did wrong. Couldn't find Again, nobody should read or write such. > anything. My only available solution was to rebuild completely my > server, something I wanted to do for a while because of other > undesired quirks that were bothering me with my setup. I knew it > would take a long time - which I didn't really have - but I finally > finished my new setup yesterday. (Don't look for > 25FC41154DCB2CAE3ABD74A8DFCD5B90D2CFFD57 or the bridge, they have > been shut down for the moment.) 3B85067588C3F017D5CCF7D8F65B5881B7D4C97C is actually running >Today, I read a line from Chris Endiku-6 saying: "Thereâs > something going on for a while and I havenât seen any mentions of > it." The exact problem I mentioned! He says it goes "as early as > Dec.23"; my problem goes to Dec 18 as shown in my previous email. > Also, not mentioned in my previous email, before I renewed my setup, > my tor-ddos firewall rules (I use the ones from Endiku-6) had blocked > about 5 times more IP than usual - if that can be useful information > to anyone. Yeah, those things are the spices in our dish. Not sure yet if this is an attack. I observe it too and investigate on my end. Trying to understand the complex vector. >I still would like to know how to restart such a relay, if this > happens again in the future - other than reinstalling the entire > server, that is. Those are my questions too :) . Case by case and issue by issue. Stay save out there! -- Cheers, Felix pgpynMp81Z0qm.pgp Description: Digitale Signatur von OpenPGP ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Relay’s first seen date got reset
This happened to several oft my nodes. They even lost their guard flags, while other nodes running on the same host are not affected. On the 5 year graph I can see their stats going back years. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] A new kind of attack?
On 1/15/24 3:19 PM, Chris Enkidu-6 wrote: I've noticed a new kind of possible attack on some of my relays, as early as Dec.23 which causes huge spikes of outbound traffic that eventually maxes out RAM and crashes Tor. The newest one today lasted for 5 hours switching between two of the three relays on the same IP. I have included charts and excerpts from the log in my post in Tor forum at below link: https://forum.torproject.org/t/new-kind-of-attack/11122 I've noticed this as well, on 0.4.8.10 across FreeBSD and Alpine platforms, against relays too new to receive any meaningful traffic from regular clients. MaxMemInQueues does not prevent the relay's eventual saturation of available memory on the system. The relays operated as exit nodes. We're low on memory (cell queues total alloc: 6336 buffer total alloc: 1556480, tor compress total alloc: 1073827425 (zlib: 0, zstd: 0, lzma: 1073827249), rendezvous cache total alloc: 0). Killing circuits│withover-long queues. (This behavior is controlled by MaxMemInQueues.) -- Jordan Savoca https://jordan.im/ ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay reboots every 15 minutes
Checking the available memory of my 2 GBytes server with Linux commend: free -m the relay runs out of memory after rebooting within a few minutes while the connections raise to the now usual 10600 connections. A few minutes the available memory is about 90 MBytes and then the relay reboots. It has been running fine for months without this memory problem. My Raspberry Pi relay with 4 GB of RAM relay does not seem to have this problem. > Op 15-01-2024 11:14 CET schreef torserver : > > > Starting about a week or so the number of connections raised rapidly to > 18000+ and since then my middle relay reboots every 15 minutes. Lowering the > relaybandwidth to a few MBytes partly solved these reboots. Before these > unplanned reboots the relay has run for months at 20 - 40 MBytes traffic > without issues. > The number of connections now is around 11000 per relay. > How can I prevent these reboots? > > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays > ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] A new kind of attack?
I've noticed a new kind of possible attack on some of my relays, as early as Dec.23 which causes huge spikes of outbound traffic that eventually maxes out RAM and crashes Tor. The newest one today lasted for 5 hours switching between two of the three relays on the same IP. During the attack, Tor becomes so busy processing the traffic that it becomes unresponsive to new connections for minutes at a time and effectively becomes a zombie exclusively processing the attacker's traffic until it eventually crashes and restarts. The interesting part is that when Tor restarts, it doesn't start from scratch building new circuits but it starts right from where it left out and keeps processing the previous connections. I have tried shutting down Tor for over 5 minutes and within one minute of restart, The RAM maxes out and the outbound traffic reaches the previous heights. This has been happening, not to all relays but to a select group of relays at a time and unless you're monitoring your Tor port from outside, you may not notice it's unresponsive. Another way to see if it's happening to you too is to check your monthly history on the metrics page and look for spikes of written bytes or sudden decrease of read bytes where you see a big gap between the two. I have included charts and excerpts from the log in my post in Tor forum at below link: https://forum.torproject.org/t/new-kind-of-attack/11122 I'd appreciate your insights and comments. Thank you. ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay that's been running for a long time suddenly saying it's new?
It's not just you. 3 of my relays show as new for the past few days and they still do. It doesn't seem to affect the traffic though so I'm assuming it's just a reporting issue and Authorities don't see your relay as new. On 1/12/2024 1:00 PM, Keifer Bly wrote: > Hi, > > So my relay > at > https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D > is suddenly saying it's a new relay. Don't know why this would happen > as it's been running for a few years, but suddenly saying it's new? > > Thanks. > > --Keifer > > ___ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay that's been running for a long time suddenly saying it's new?
I had this issue too. It resolved itself shortly within a few hours. Original Message On Jan 15, 2024, 05:23, Petrarca via tor-relays wrote: > Just to confirm - the same happens to my relay, so this seems to be a general > issue. > > Keifer Bly schrieb am Montag, 15. Januar 2024 um 09:29: > >> Hi, >> >> So my relay at >> https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D >> is suddenly saying it's a new relay. Don't know why this would happen as >> it's been running for a few years, but suddenly saying it's new? >> >> Thanks. >> >> --Keifer___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay that's been running for a long time suddenly saying it's new?
Just to confirm - the same happens to my relay, so this seems to be a general issue. Keifer Bly schrieb am Montag, 15. Januar 2024 um 09:29: > Hi, > > So my relay at > https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D > is suddenly saying it's a new relay. Don't know why this would happen as > it's been running for a few years, but suddenly saying it's new? > > Thanks. > > --Keifer___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay that's been running for a long time suddenly saying it's new?
There seems to be an issue with the metrics page, mine(3A8D61AC59FD4F9AC7CC82B4B58FCC451578DC3B) has higher uptime than the first seen which is very interesting 樂 \ Original Message On Jan 15, 2024, 1:33 PM, Keifer Bly < keifer@gmail.com> wrote: > > > > Hi, > > > > > So my relay at > [https://metrics.torproject.org/rs.html\#details/79E3B585803DE805CCBC00C1EF36B1E74372861D][https_metrics.torproject.org_rs.html_details_79E3B585803DE805CCBC00C1EF36B1E74372861D] > is suddenly saying it's a new relay. Don't know why this would happen as > it's been running for a few years, but suddenly saying it's new? > > > > > Thanks. > > > > > \--Keifer [https_metrics.torproject.org_rs.html_details_79E3B585803DE805CCBC00C1EF36B1E74372861D]: https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D publickey - EmailAddress(s=tor@szaboaleks.xyz) - 0x2A931C00.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Relay reboots every 15 minutes
Starting about a week or so the number of connections raised rapidly to 18000+ and since then my middle relay reboots every 15 minutes. Lowering the relaybandwidth to a few MBytes partly solved these reboots. Before these unplanned reboots the relay has run for months at 20 - 40 MBytes traffic without issues. The number of connections now is around 11000 per relay. How can I prevent these reboots? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Re: [tor-relays] Relay that's been running for a long time suddenly saying it's new?
Keifer Bly: Hi, So my relay at https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D is suddenly saying it's a new relay. Don't know why this would happen as it's been running for a few years, but suddenly saying it's new? Should be fine again right now, no? We've experiencing issues like that for some days now and they are very likely somehow related to CollecTor issues we have, which are tracked at https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/issues/40038 Hope this helps, Georg Thanks. --Keifer ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays OpenPGP_signature.asc Description: OpenPGP digital signature ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Tor relay operator research
Dear operators, I'm conducting a study to understand relay operator motivations and needs, funded by OTF. If you had about 30-45 minutes to talk (completely anonymously if you wish) about your experience as a relay operator, it will really help this research, especially if you're running a relay in non-western country. If interested please reach out to me either on ac...@torproject.org or a...@sr2.uk. Cheers Ana ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Recent Tor versions not reloading config on / ignoring HUP kill signal.
Hi,
I think this started with release 0.4.8.10, but both of my Tor relays no longer
reload their config when doing for example:
- systemctl reload tor@exit
Here is the relevant part of the unit file:
> [Unit]Description=Anonymizing overlay network for TCP
> After=syslog.target network.target nss-lookup.target
>
> [Service]
> Type=notify
> NotifyAccess=all
> ExecStartPre=/usr/bin/tor -f /etc/tor/torrc_%i --verify-config
> ExecStart=/usr/bin/tor -f /etc/tor/torrc_%i
> ExecReload=/bin/kill -HUP ${MAINPID}
> KillSignal=SIGINT
> TimeoutSec=75
> Restart=on-failure
> WatchdogSec=1m
> LimitNOFILE=32768
Checking with:
- journalctl -u tor@exit
Just tells me that systemd attempted and successfully executed the specified
reload command, but the actual line from the Tor instance stating that the
config has been reloaded is missing.
Is anyone else experiencing this?
Regards,
George
publickey - hartley_george@proton.me - 0xAEE8E00F.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
___
tor-relays mailing list
tor-relays@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Relay’s first seen date got reset
I run the following relay: https://metrics.torproject.org/rs.html#details/6C336E553CC7E0416EBC8577A7289349B757F6C3. I just noticed that my relay’s ‘first seen’ date got reset. Tor now thinks that my relay is less than 2 weeks old. But when you open the 6 months graph, you can see the actual ‘first seen’ date which is November 29th 2023. Is it possible to fix this ‘first seen’ date back to the actual value? ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
[tor-relays] Relay that's been running for a long time suddenly saying it's new?
Hi, So my relay at https://metrics.torproject.org/rs.html#details/79E3B585803DE805CCBC00C1EF36B1E74372861D is suddenly saying it's a new relay. Don't know why this would happen as it's been running for a few years, but suddenly saying it's new? Thanks. --Keifer ___ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
