Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
-+-
 Reporter:  arma |  Owner:  dgoulet
 Type:  defect   | Status:  closed
 Priority:  Medium   |  Milestone:  Tor:
 |  0.4.3.x-final
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  network-health, 043-should,  |  Actual Points:
  security, 044-must |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by nickm):

 * status:  assigned => closed
 * resolution:   => fixed
 * milestone:  Tor: 0.4.4.x-final => Tor: 0.4.3.x-final


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+--
 Reporter:  arma|  Owner:  dgoulet
 Type:  defect  | Status:  assigned
 Priority:  Medium  |  Milestone:  Tor:
|  0.4.4.x-final
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health 043-should security  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by asn):

 * keywords:  network-health 043-should => network-health 043-should
   security


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by ln5):

 And here's dizum's numbers.

 {{{
  day rx  | tx  |total|   avg. rate
  +-+-+---
  02/02/202050.36 GiB |2.22 TiB |2.27 TiB |  231.50 Mbit/s
  02/03/202052.95 GiB |2.30 TiB |2.35 TiB |  239.51 Mbit/s
  02/04/202046.97 GiB |1.95 TiB |2.00 TiB |  203.64 Mbit/s
  02/05/202037.35 GiB |1.27 TiB |1.30 TiB |  132.51 Mbit/s
  02/06/202026.98 GiB |  974.02 GiB |0.98 TiB |   99.52 Mbit/s
  02/07/202020.72 GiB |  608.53 GiB |  629.25 GiB |   62.56 Mbit/s
  02/08/202019.12 GiB |  514.00 GiB |  533.12 GiB |   53.00 Mbit/s
  02/09/202017.44 GiB |  473.57 GiB |  491.01 GiB |   48.82 Mbit/s
  02/10/202030.49 GiB |  656.36 GiB |  686.85 GiB |   68.29 Mbit/s
  02/11/202028.25 GiB |  646.54 GiB |  674.78 GiB |   67.09 Mbit/s
  02/12/202015.40 GiB |  536.50 GiB |  551.90 GiB |   54.87 Mbit/s
  02/13/202015.99 GiB |  567.34 GiB |  583.33 GiB |   58.00 Mbit/s
  02/14/202015.39 GiB |  541.78 GiB |  557.17 GiB |   55.39 Mbit/s
  02/15/202013.90 GiB |  443.84 GiB |  457.74 GiB |   45.51 Mbit/s
  02/16/202013.64 GiB |  401.60 GiB |  415.24 GiB |   41.28 Mbit/s
  02/17/202015.20 GiB |  535.51 GiB |  550.70 GiB |   54.75 Mbit/s
  02/18/202010.61 GiB |  183.93 GiB |  194.54 GiB |   19.34 Mbit/s
  02/19/202010.77 GiB |  208.25 GiB |  219.02 GiB |   21.78 Mbit/s
  02/20/202011.70 GiB |  273.64 GiB |  285.35 GiB |   28.37 Mbit/s
  02/21/202011.31 GiB |  230.19 GiB |  241.50 GiB |   24.01 Mbit/s
  02/22/2020 9.76 GiB |  157.14 GiB |  166.89 GiB |   16.59 Mbit/s
  02/23/2020 9.66 GiB |  154.74 GiB |  164.40 GiB |   16.34 Mbit/s
  02/24/202010.21 GiB |  169.80 GiB |  180.01 GiB |   17.90 Mbit/s
  02/25/2020 9.55 GiB |  156.26 GiB |  165.81 GiB |   16.48 Mbit/s
  02/26/2020 9.85 GiB |  157.76 GiB |  167.61 GiB |   16.66 Mbit/s
  02/27/202011.43 GiB |  165.83 GiB |  177.25 GiB |   17.62 Mbit/s
  02/28/202016.74 GiB |  369.09 GiB |  385.84 GiB |   38.36 Mbit/s
  02/29/202028.31 GiB |1.45 TiB |1.48 TiB |  150.25 Mbit/s
  03/01/202031.82 GiB |1.81 TiB |1.84 TiB |  187.09 Mbit/s
  03/02/202012.73 GiB |  763.21 GiB |  775.94 GiB |  211.82 Mbit/s
 }}}

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by ln5):

 Here's the amount of traffic maatuska has seen since 2020-01-21 according
 to vnstat:

 {{{
   dayrx  | tx  |total|   avg. rate
  +-+-+---
  2020-01-21 7.26 GiB |  334.60 GiB |  341.86 GiB |   33.99 Mbit/s
  2020-01-2225.87 GiB |1.02 TiB |1.04 TiB |  105.96 Mbit/s
  2020-01-2320.76 GiB |  725.33 GiB |  746.09 GiB |   74.18 Mbit/s
  2020-01-2428.96 GiB |1.16 TiB |1.19 TiB |  121.02 Mbit/s
  2020-01-2547.43 GiB |2.13 TiB |2.18 TiB |  222.03 Mbit/s
  2020-01-2646.79 GiB |2.13 TiB |2.18 TiB |  221.50 Mbit/s
  2020-01-2749.37 GiB |2.24 TiB |2.29 TiB |  232.67 Mbit/s
  2020-01-2855.66 GiB |2.58 TiB |2.63 TiB |  267.96 Mbit/s
  2020-01-2955.63 GiB |2.47 TiB |2.52 TiB |  256.57 Mbit/s
  2020-01-3059.04 GiB |2.53 TiB |2.59 TiB |  263.87 Mbit/s
  2020-01-3157.38 GiB |2.43 TiB |2.49 TiB |  253.32 Mbit/s
  2020-02-0159.35 GiB |2.29 TiB |2.35 TiB |  239.08 Mbit/s
  2020-02-0262.08 GiB |2.25 TiB |2.31 TiB |  235.25 Mbit/s
  2020-02-0362.58 GiB |2.31 TiB |2.37 TiB |  241.57 Mbit/s
  2020-02-0454.02 GiB |1.94 TiB |1.99 TiB |  202.94 Mbit/s
  2020-02-0542.82 GiB |1.25 TiB |1.29 TiB |  131.31 Mbit/s
  2020-02-0630.12 GiB |  948.48 GiB |  978.60 GiB |   97.29 Mbit/s
  2020-02-0721.90 GiB |  573.29 GiB |  595.18 GiB |   59.17 Mbit/s
  2020-02-0820.29 GiB |  491.28 GiB |  511.57 GiB |   50.86 Mbit/s
  2020-02-0918.50 GiB |  446.70 GiB |  465.20 GiB |   46.25 Mbit/s
  2020-02-1032.86 GiB |  640.45 GiB |  673.31 GiB |   66.94 Mbit/s
  2020-02-1130.74 GiB |  627.65 GiB |  658.39 GiB |   65.46 Mbit/s
  2020-02-1217.11 GiB |  514.04 GiB |  531.15 GiB |   52.81 Mbit/s
  2020-02-1317.23 GiB |  541.86 GiB |  559.09 GiB |   55.58 Mbit/s
  2020-02-1417.12 GiB |  527.92 GiB |  545.05 GiB |   54.19 Mbit/s
  2020-02-1515.40 GiB |  433.57 GiB |  448.97 GiB |   44.64 Mbit/s
  2020-02-1614.84 GiB |  390.59 GiB |  405.43 GiB |   40.31 Mbit/s
  2020-02-1717.35 GiB |  526.00 GiB |  543.36 GiB |   54.02 Mbit/s
  2020-02-1810.97 GiB |  156.47 GiB |  167.44 GiB |   16.65 Mbit/s
  2020-02-1911.46 GiB |  185.11 GiB |  196.57 GiB |   19.54 Mbit/s
  2020-02-2012.69 GiB |  261.25 GiB |  273.94 GiB |   27.24 Mbit/s
  2020-02-2112.25 GiB |  217.21 GiB |  229.46 GiB |   22.81 Mbit/s
  2020-02-2211.21 GiB |  161.49 GiB |  172.70 GiB |   17.17 Mbit/s
  2020-02-2310.98 GiB |  155.61 GiB |  166.59 GiB |   16.56 Mbit/s
  2020-02-2410.96 GiB |  158.25 GiB |  169.21 GiB |   16.82 Mbit/s
  2020-02-2510.21 GiB |  141.04 GiB |  151.26 GiB |   15.04 Mbit/s
  2020-02-2610.51 GiB |  148.31 GiB |  158.82 GiB |   15.79 Mbit/s
  2020-02-2710.83 GiB |  157.55 GiB |  168.38 GiB |   16.74 Mbit/s
  2020-02-2814.19 GiB |  355.08 GiB |  369.27 GiB |   36.71 Mbit/s
  2020-02-2933.92 GiB |1.47 TiB |1.51 TiB |  153.29 Mbit/s
  2020-03-0140.64 GiB |1.84 TiB |1.88 TiB |  191.82 Mbit/s
  2020-03-0216.34 GiB |  796.83 GiB |  813.17 GiB |  215.59 Mbit/s
 }}}

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by starlight):

 How about this?  A hashlimit rule that establishes a maximum byte rate out
 to any /24, /22 or such.  My point is netfilter allows rapid what-if
 testing of various rate limit strategies, up-to and including HTB ingress
 schemes with IMQ and arbitrary schemes with nfqueue.  Dirport queries are
 clear-text, it should be manageable to distinguish between compressed and
 uncompressed queries and mark connections accordingly.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by starlight):

 Consensus documents are large and I suppose a fairly low rate per address,
 coming from a few thousand would result in overload. . .was just a
 thought.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by arma):

 Replying to [comment:19 starlight]:
 > How about throwing up an iptables recent module rule limiting the
 maximum request rate from any single IP address and seeing if it helps?
 My observation is botnet abuse often arrives with high intensity from a
 limited number of addresses during a given interval.  Set at a perhaps
 twice the maximum rate expected of a bootstrapping relay.  If it works
 similar logic could be added to the relay.  I have a couple of rules I'll
 share privately, though it's hardly rocket science.

 I believe Sebastian has done some exploration of the IP addresses, and
 found that many of the requests come from their own IP address. That is,
 this really is thousands of places around the internet, not just a
 handful.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by starlight):

 How about throwing up an iptables recent module rule limiting the maximum
 request rate from any single IP address and seeing if it helps?  My
 observation is botnet abuse often arrives with high intensity from a
 limited number of addresses during a given interval.  Set at a perhaps
 twice the maximum rate expected of a bootstrapping relay.  If it works
 similar logic could be added to the relay logic.  I have a couple of rules
 I'll share privately, though it's hardly rocket science.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by cypherpunks):

 Replying to [comment:13 arma]:
 > Replying to [comment:3 cypherpunks]:
 > > In the Metrics the increase is not seen?
 > > http://rougmnvswfsmd4dq.onion/dirbytes.html
 >
 > Thanks. I've filed #33065 to investigate.
 Thanks too.

 It shows increase around one million new directly connecting users within
 first month of 2020.

 http://rougmnvswfsmd4dq.onion/userstats-relay-
 country.html?start=2019-12-01=2020-01-29=all=off
 I regulary checked Graphs and may this increase  is somehow related?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by arma):

 I sent a status update to the dir-auths, so they can run the simplified
 #33029 patch while we work on better fixes:
 https://lists.torproject.org/pipermail/network-
 health/2020-January/000457.html

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by teor):

 Replying to [comment:12 Sebastian]:
 > Replying to [comment:8 teor]:
 > > Replying to [comment:2 Sebastian]:
 > > > I am not concerned about relays connecting from a wrong IP address.
 I basically feel like that shouldn't even be possible configuration-wise
 > >
 > > Relays can set different addresses in the Address and
 OutboundBindAddress options, and their inbound and outbound traffic will
 be on different addresses. Some operators use these options, others put
 their Address on a non-default route.
 > >
 > > So we do need to consider this case, particularly when relays are
 trying to discover their own IP address from an authority. But relays
 should fall back to discovering their address and getting a consensus from
 other relays, if all the authorities fail.
 > >
 > > So maybe it will work anyway? We should do a test to make sure.
 >
 > I know these kinds of configurations are possible, but why is that and
 why are we OK with it. That's my point here, we should IMO change your
 stance to this being not supported behaviour.

 At the moment, relays (and directory authorities) use the system default
 route for outbound traffic, rather than the advertised address.

 If you want to change OutboundBindAddress, here's what we could do:
 1. make OutboundBindAddress default to the advertised addresses (IPv4 and
 IPv6), but fall back to unbound if binding to a specific address doesn't
 work - fixes most relays
 2. deprecate OutboundBindAddress - needs proposal (or consultation with
 relay operators), fixes some of the rest

 We can't fix all the relays, because operators can still use firewalls
 (and other weird network configs) to change the outbound address.

 I'll put step 1 in my upcoming IPv6 address discovery proposal as optional
 work. I think it's best we do step 2 separately. Because it's likely to be
 controversial.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by nickm):

 I agree that #33029 is a defense that we should have on tap to handle this
 kind of problem going forward.  It's not perfect, so maybe there should be
 an option to enable or disable it.

 Sebastian has also offered to run a patch for me that will record a sample
 of HTTP GET headers.  I'm hoping to find a pattern that we can detect.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  dgoulet
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---
Changes (by ahf):

 * status:  new => assigned
 * owner:  (none) => dgoulet


Comment:

 Assigning this with David as owner during the 0.4.3 IRC ticket triage.
 Nick notes here that he will happily help out as well and others can join
 in as well.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  (none)
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by dgoulet):

 Published a new version of the code on the PR.

 It considers IPv6 but as teor mentions, only the trusted configured
 authorities, not from the consensus. For dirauth, that is fine for now
 since they all configure the other dirauths in their torrc.

 Also, this can backfire if a dirauth outbound address is different from
 its inbound DirPort address.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  (none)
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by arma):

 Replying to [comment:3 cypherpunks]:
 > In the Metrics the increase is not seen?
 > http://rougmnvswfsmd4dq.onion/dirbytes.html

 Thanks. I've filed #33065 to investigate.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  (none)
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---

Comment (by Sebastian):

 Replying to [comment:8 teor]:
 > Replying to [comment:2 Sebastian]:
 > > I am not concerned about relays connecting from a wrong IP address. I
 basically feel like that shouldn't even be possible configuration-wise
 >
 > Relays can set different addresses in the Address and
 OutboundBindAddress options, and their inbound and outbound traffic will
 be on different addresses. Some operators use these options, others put
 their Address on a non-default route.
 >
 > So we do need to consider this case, particularly when relays are trying
 to discover their own IP address from an authority. But relays should fall
 back to discovering their address and getting a consensus from other
 relays, if all the authorities fail.
 >
 > So maybe it will work anyway? We should do a test to make sure.

 I know these kinds of configurations are possible, but why is that and why
 are we OK with it. That's my point here, we should IMO change your stance
 to this being not supported behaviour.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
---+---
 Reporter:  arma   |  Owner:  (none)
 Type:  defect | Status:  new
 Priority:  Medium |  Milestone:  Tor:
   |  0.4.3.x-final
Component:  Core Tor/Tor   |Version:
 Severity:  Normal | Resolution:
 Keywords:  network-health 043-should  |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+---
Changes (by nickm):

 * keywords:  network-health => network-health 043-should
 * milestone:   => Tor: 0.4.3.x-final


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by teor):

 Some of these changes are significant, and they are going to have impacts
 across the network. But I understand that we need some of these fixes
 urgently.

 Can you please write proposals for the longer-term changes?
 And send a quick email to tor-dev about the urgent changes?

 I'm concerned that there may be design conflicts with some other work,
 particularly Sponsor 55. So we need to share these details with other tor
 developers. That way, we can make sure that the Sponsor 55 proposals are
 compatible with these fixes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by teor):

 Replying to [comment:5 arma]:
 > Possible next steps beyond the above branch which I think would be worth
 taking:
 >
 > 1. Whitelist (i.e. never send 503's) IP addresses of relays in the
 consensus too. Or maybe it's better to consider relays in our descriptor
 list (i.e. if we vote about it, whitelist it). I have a commented-out
 function conn_addr_is_relay() in the above branch which somebody would
 need to write, and it will need to be fast fast fast or the lookup won't
 be worth it. ahf sketched out that function as "if we extend routerlist_t
 to have a map from addr to a routerinfo_t and from the v6 address, then I
 think you can do it fast."

 It's better to consider the descriptor list, otherwise new relays have
 trouble joining the consensus. It's important to whitelist IPv4 and IPv6
 addresses, because Sponsor 55 will add IPv6 directory requests.

 > 2. Whitelist the IP address for the consensus health checker (I think
 that might be carinatum.tpo) so it stops yelling and thinking we're down.
 :)
 >
 > 3. Consider giving higher priority to microdesc-consensus and microdesc
 replies. That is, I would rather have relays successfully cache and mirror
 the microdesc flavored stuff, if I have to choose.
 >
 > 4. Make a change to the Tor code so relays remain on the client fetch
 schedule (i.e. fetch from relays and fallback dirs) until they publish
 their descriptor. That way we remove one variable from the mystery, i.e.
 "maybe these Tors that are mobbing me are all configured as relays but
 haven't found themselves reachable so that's why I don't know about them."
 (I recognize we'll need to wait some years until everybody has upgraded.
 No time like the present to get started then.)

 Relays which don't know their address will fetch from authorities, so they
 get their address from a trusted source. But address discovery only needs
 one successful fetch (or two, once relays are trying to guess their IPv6
 address). After that, relays can use the client fetch targets.

 The client schedule is a slightly different thing, it only affects fetch
 timing. It's ok for relays to use the client fetch timing, until they
 publish their descriptor. Relays on bad links might bootstrap a bit more
 slowly or unreliably, but those relays were never going to be good relays
 anyway.

 And then there's the client and relay fetch method. Should relays use
 ORPorts for fetches until they publish their descriptor? It probably
 wouldn't hurt, and it would make address detection more secure.

 > 5. Look for patterns in the non-relay IP addresses that are bombing us
 with consensus fetch attempts. How often do they come back asking for
 another one? Does that timing pattern make us think they are a well
 behaving Tor that somehow thinks the dir auths' dirports are the best
 places to ask?

 We should also check the HTTP headers sent as part of the requests. They
 will tell us a lot about the tor version (or other program) that's sending
 the requests.

 > 6. Consider a design for a more aggressive load shedding plan. Right now
 we send the 503 if we don't have the space left in our global write
 bucket, or we ran out of global write bucket the previous second. For
 vanilla-flavored dirport consensus responses to non-relay IP addresses, I
 could imagine something much more aggressive, like "could I serve ten of
 these? No? Then 503." with the goal of actually leaving some room to serve
 the more important ones rather than always being full or nearly full.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by teor):

 Replying to [comment:2 Sebastian]:
 > I am not concerned about relays connecting from a wrong IP address. I
 basically feel like that shouldn't even be possible configuration-wise

 Relays can set different addresses in the Address and OutboundBindAddress
 options, and their inbound and outbound traffic will be on different
 addresses. Some operators use these options, others put their Address on a
 non-default route.

 So we do need to consider this case, particularly when relays are trying
 to discover their own IP address from an authority. But relays should fall
 back to discovering their address and getting a consensus from other
 relays, if all the authorities fail.

 So maybe it will work anyway? We should do a test to make sure.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by arma):

 dgoulet has a patch for (1) in #33029.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by dgoulet):

 About (1), the DoS subsystem added a way to identify if the address is
 "probably" a relay or not in a very fast way with a bloom filter. Relay
 address are added when a `node_t` is added to the routerlist which for a
 dirauth contains all relays that have uploaded their descriptors.

 Code is in `./src/core/or/address_set.h` which is very convenient and
 already in use.

 We could implement (1) easily using this.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by arma):

 Possible next steps beyond the above branch which I think would be worth
 taking:

 * Whitelist (i.e. never send 503's) IP addresses of relays in the
 consensus too. Or maybe it's better to consider relays in our descriptor
 list (i.e. if we vote about it, whitelist it). I have a commented-out
 function conn_addr_is_relay() in the above branch which somebody would
 need to write, and it will need to be fast fast fast or the lookup won't
 be worth it. ahf sketched out that function as "if we extend routerlist_t
 to have a map from addr to a routerinfo_t and from the v6 address, then I
 think you can do it fast."

 * Whitelist the IP address for the consensus health checker (I think that
 might be carinatum.tpo) so it stops yelling and thinking we're down. :)

 * Consider giving higher priority to microdesc-consensus and microdesc
 replies. That is, I would rather have relays successfully cache and mirror
 the microdesc flavored stuff, if I have to choose.

 * Make a change to the Tor code so relays remain on the client fetch
 schedule (i.e. fetch from relays and fallback dirs) until they publish
 their descriptor. That way we remove one variable from the mystery, i.e.
 "maybe these Tors that are mobbing me are all configured as relays but
 haven't found themselves reachable so that's why I don't know about them."

 * Look for patterns in the non-relay IP addresses that are bombing us with
 consensus fetch attempts. How often do they come back asking for another
 one? Does that timing pattern make us think they are a well behaving Tor
 that somehow thinks the dir auths' dirports are the best places to ask?

 * Consider a design for a more aggressive load shedding plan. Right now we
 send the 503 if we don't have the space left in our global write bucket,
 or we ran out of global write bucket the previous second. For vanilla-
 flavored dirport consensus responses to non-relay IP addresses, I could
 imagine something much more aggressive, like "could I serve ten of these?
 No? Then 503." with the goal of actually leaving some room to serve the
 more important ones rather than always being full or nearly full.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by arma):

 My {{{ticket33018-dirauth-503}}} branch resumes being willing to send 503
 responses to dir requests, even for dir auths, but it avoids sending a 503
 when the requesting IP address is itself a dir auth.

 I'm running it on moria1 now, and I tested it with log_notice lines to
 make sure it correctly recognizes requests from other dir auths.

 I don't actually intend this patch to be merged into mainline Tor -- it is
 instead a demo hack of one way forward here.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by cypherpunks):

 In the Metrics the increase is not seen?
 http://rougmnvswfsmd4dq.onion/dirbytes.html

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by Sebastian):

 I am not concerned about relays connecting from a wrong IP address. I
 basically feel like that shouldn't even be possible configuration-wise

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #33018 [Core Tor/Tor]: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix

[Tor Bug Tracker & Wiki]

#33018: Dir auths using an unsustainable 400+ mbit/s, need to diagnose and fix
+
 Reporter:  arma|  Owner:  (none)
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Core Tor/Tor|Version:
 Severity:  Normal  | Resolution:
 Keywords:  network-health  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+

Comment (by arma):

 Initial impressions: these are requests to the DirPort (not the ORPort),
 and they're coming from many different IP addresses, most of which are not
 current relay IP addresses.

 I had 1000+ connections to my DirPort in TCP state ESTABLISHED, and kill
 -USR1 said
 {{{
 For 1139 Directory connections: 43795985 used/48406528 allocated
 }}}
 i.e. at that moment I had already committed to answering 43megabytes of
 dir info that I hadn't managed to push onto the network yet.

 Most requests seem to be for "/tor/status-vote/current/consensus" which is
 the vanilla-flavored consensus, not the microdesc-flavored consensus that
 is actually in use by clients.

 It would be useful for Tor to collect statistics about how many requests,
 and how many bytes, were for what sort of dir object, and came from relay
 vs non-relay IP addresses.

 Another idea for an improvement is that we might change Tors to only fetch
 from the dir auths once they have decided to publish their relay
 descriptor, i.e. if you are a relay but you are not reachable, you should
 stay on the "client" fetch schedule. That way it is easier to say that if
 you are fetching from moria1 but you are not a relay, it is surprising and
 weird. (Still a bit tricky though, because relays might connect to
 moria1's dirport from a different IP address than they write in their
 descriptor.)

 Also, handle_get_current_consensus() checks
 {{{
   if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) {
 log_debug(LD_DIRSERV,
   "Client asked for network status lists, but we've been "
   "writing too many bytes lately. Sending 503 Dir busy.");
 }}}

 but global_write_bucket_low() says
 {{{
   if (authdir_mode(get_options()) && priority>1)
 return 0; /* there's always room to answer v2 if we're an auth dir */
 }}}

 I have commented these lines out on moria1, and now I am sending dozens of
 503 responses per second. This is sort of sad for legit relays that want
 to get their answers, but I think it should make bandwidth available to
 other directory operations.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs