On Sun, 26 Mar 2023 17:17:25 -0700 Mike Lyon <[email protected]> wrote:
> Can’t believe it’s still dead… > > -Mike The attack appears to be over, at Mar 26 13:41:28 JST (GMT +0900) (This may be specific to my server). Maybe the cause is something else. Or the person in charge of manual recovery is on holiday. Mar 26 13:41:08 unbound[48103:0] reply: 24.199.82.210 asm.faa.gov. A IN SERVFAIL 0.000000 0 29 Mar 26 13:41:15 unbound[48103:0] query: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:15 unbound[48103:0] reply: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN SERVFAIL 0.000000 0 41 Mar 26 13:41:22 unbound[48103:0] query: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 Mar 26 13:41:23 unbound[48103:0] query: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 Mar 26 13:41:28 unbound[48103:0] query: 24.199.82.210 chronos3.faa.gov. A IN Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34 > > On Mar 26, 2023, at 17:13, T.Suzuki via Outages <[email protected]> wrote: > > > > 〓On Sun, 26 Mar 2023 08:35:29 -0700 > > Hugo Slabbert <[email protected]> wrote: > > > >> What would be the symptoms here of a "water torture attack" rather than > >> what John had indicated as a firewall failure in their infrastructure: > >> > >>> Initial looks from the firewall team point to an automatic failover event > >> and the secondary failed. > >> > >> And the symptoms of which lined up with network level info from Paul > >> earlier: > >> > >>> They only seem to have two auth nameservers for faa, both within the > >> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the > >> servers are in all die just within each block run by the FAA. > >>> > >>> Seems like an internal routing meltdown making the only 2 nameservers > >> unreachable reliably. > >> > >> Are you saying that your open resolvers have a per client rate limit > >> applied, that rate limit got tripped, and shortly thereafter the resolvers > >> became unavailable, suggesting query floods for the domain(s) that knocked > >> the resolvers offline (or from the other discussion, possibly was the thing > >> that overwhelmed that firewall layer, causing the initial failover and > >> possibly also causing the firewall secondary to fail to come online)? > > > > Yes. (limitting per client, and per second for all) > > Perhaps, large numbers open resolvers including no ratelimit are used. > > Then massive random subdomain queries caused the firewall symptoms. > > (It's only my guess.) > > > >>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <[email protected]> > >>> wrote: > >>> > >>> Hi, I'm a researcher of DNS vulnerabilities. > >>> > >>> It loos like random subdomain attacks (water tourtue attack). > >>> > >>> This is the data of my rate-limitted openresolver as a honeypot. > >>> http://www.e-ontap.com/dns/todaydowngov.txt > >>> http://www.e-ontap.com/dns/todaydown.txt > >>> (You can not view these page if you are using 8.8.8.8, sorry.) > >>> > >>> Raw logs of my Unbound (Time is JST) > >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head > >>> -5 > >>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL < > >>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov. > >>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 > >>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45 > >>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: > >>> exceeded ratelimit for zone faa.gov. > >>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN > >>> SERVFAIL 15.112813 0 30 > >>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: > >>> exceeded ratelimit for zone faa.gov. > >>> local/etc/unbound% > >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | > >>> head -5 > >>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: > >>> all servers for this domain failed, at zone faa.gov. from > >>> 2620:74:27::2:30 no server to query nameserver addresses not usable > >>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL > >>> <lyndas365project.faa.gov. > >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to > >>> query nameserver addresses not usable > >>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all > >>> servers for this domain failed, at zone faa.gov. no server to query > >>> nameserver addresses not usable > >>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: > >>> all servers for this domain failed, at zone faa.gov. upstream server > >>> timeout > >>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. > >>> A IN>: all servers for this domain failed, at zone faa.gov. upstream > >>> server timeout > >>> local/etc/unbound% > >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | > >>> tail -5 > >>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all > >>> servers for this domain failed, at zone faa.gov. no server to query > >>> nameserver addresses not usable > >>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. > >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to > >>> query nameserver addresses not usable > >>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL < > >>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at > >>> zone faa.gov. no server to query nameserver addresses not usable > >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < > >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, > >>> at zone faa.gov. no server to query nameserver addresses not usable > >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A > >>> IN>: all servers for this domain failed, at zone faa.gov. no server to > >>> query nameserver addresses not usable > >>> local/etc/unbound% > >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail > >>> -5 > >>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 > >>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44 > >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL < > >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, > >>> at zone faa.gov. no server to query nameserver addresses not usable > >>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 > >>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46 > >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A > >>> IN>: all servers for this domain failed, at zone faa.gov. no server to > >>> query nameserver addresses not usable > >>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A > >>> IN SERVFAIL 0.000000 0 34 > >>> local/etc/unbound% > >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc > >>> -l > >>> 1408 > >>> > >>> -- > >>> T.Suzuki > >>> -- > >>> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう > >>> _______________________________________________ > >>> Outages mailing list > >>> [email protected] > >>> https://puck.nether.net/mailman/listinfo/outages > >>> > > > > > > -- > > T.Suzuki / E.F.シューマッハーとI.イリイチを読もう > > _______________________________________________ > > Outages mailing list > > [email protected] > > https://puck.nether.net/mailman/listinfo/outages > -- T.Suzuki / E.F.シューマッハーとI.イリイチを読もう _______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
