#14006: Hidden service error: "We'd like to launch a circuit to handle a connection, but we already have 32 general-purpose client circuits..." -----------------------------------------------+--------------------------- Reporter: asn | Owner: (none) Type: defect | Status: | needs_information Priority: Medium | Milestone: Tor: | 0.3.3.x-final Component: Core Tor/Tor | Version: Severity: Normal | Resolution: Keywords: tor-hs circuit-management scaling | Actual Points: Parent ID: | Points: Reviewer: | Sponsor: -----------------------------------------------+---------------------------
Comment (by dgoulet): Replying to [comment:20 asn]: > I think what I would prefer here is for Tor to rate-limit itself when building onion service circuits. Especially so when it has multiple onion services, but maybe even when it has only a single one. So instead of building all its onion circuits (IPs + hsdir circs) at once, it waits a randomized time (around a second?) before building each one. The problem with adding a random delay at startup is that it won't solve the "32 general purpose circuits are pending" issue. If those circuits a really stuck being built, the delay won't help much as they will all end up queued up and stuck at some point. A wise rate limit is probably what we want so we never go above that 32 limit and thus no need for a cryptic warning that makes it that you just can't do anything but wait or/and panic. Now ok, looking a bit closely to the logs above, notice: {{{ Jan 21 10:53:40.000 [warn] Giving up launching first hop of circuit to rendezvous point $9844B981A80B3E4B50897098E2D65167E6AEF127~$9844B981A80B3E4B50 at 62.138.7.171 for service eb3w4t..... }}} The above is a service trying to open a circuit to a rendezvous point... So I think the bigger issue here is that we have 32 circuits stuck in a non OPEN state and just never expire for some reasons? Or they do but we open 32 new ones very quickly and they get stalled again in a non OPEN state. My money is on the later due to the *amount* of suppressed log (see below). This looks to me like a service getting a ridiculous amount of rendezvous requests, the Guard is chocking so we keep reaching that 32 limit. {{{ Jan 21 10:37:42.000 [notice] We'd like to launch a circuit to handle a connection, but we already have 32 general-purpose client circuits pending. Waiting until some finish. [215959 similar message(s) suppressed in last 600 seconds] }}} Quick skim, I don't see anything in `circuit_expire_building()` that would make a circuit be ignored in the GUARD_WAIT state so they should in theory expire even though they are waiting for the guard to be usable? I'm getting indeed more and more convinced that we need a rate limit both client and service side. That is a bit like we do with DoS mitigation now (#24902) which is some per second rate with burst. Busy hidden service will suffer reachability but at least won't break the network. The point is that DoS mitigation will prevent as much as possible a client DDoS towards a single service and that service will by itself prevent to DDoS the network. -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14006#comment:21> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online
_______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs