OK, so I did a few test results from flent-atlanta.taht.net. They are here:
http://flent-atlanta.taht.net/~d/atl.tgz Trying to run it at close to a gbit, there are all sorts of interesting behavior between the two dcs. Using sqm + sch_fq I got mildly saner results, but... well, download the above and scratch your head some.... On Sun, Feb 9, 2020 at 8:31 AM Dave Taht <[email protected]> wrote: > > > I note that I totally missed this thread somehow. > > I have about 12 (rather undermaintained) servers in linode's cloud these > days that have netperf and irtt on them, all over the world, singapore, > mumbai, etc, etc.... > > The monthly transfer allotment is ~28TB across them all. So far this > month we've used up ~1.2TB, principally from flent-fremont.bufferbloat.net. > > The current cost, which includes bufferbloat.net and taht.net, is > 180/month. I just vector a portion of the donations to patreon over > there and don't think about it much. All of these boxes could use love > and an upgrade (softwarewise), especially bufferbloat.net, which could > also get a downgrade cpuwise. The investment of time (days) to do that, > vs the ongoing extra cost of a 2 cpu machine (15/dollars/month) has not > seemed worth it. > > I would like it very much if we were more serious about a published > reliable test infrastructure for flent. However, it was also my hope > that flent would get mostly used inside people's firewalls on their own > testbeds while developing new gear and drivers and network designs, as > it's so easy to do so (apt-get install irtt netperf flent etc) nowadays. > > One of my concerns has long been that I have no idea what lies > underneath these cloudy vms (anywhere) that is actually doing the rate > limiting (anyone? AWS? linode?), and have kind of wished we could go > back to at least one piece of bare metal on the internet as we had when > isc.org was still donating services to us. Back then vm jitter was > measured in the 10s of ms using xen... and we have no means to compare > vm performance vs bare metal performance with all this finicy stuff, > such as pacing, SCE/L4S etc. The google folk are perpetually publishing > ce_threshold set at 260us, which is just impossible (IMHO) in a vm.... > > The "nanodes" in this cloud are limited to 100Mbit, somehow. > > I've occasionally priced out what it would take to get a 10Gbit in > hurricane's cages in california - 900/month just for transit when on > special, 2k a month otherwise.... Half a cage was another 400 or so. I > already have a spare /24 and BGP AS and hardware I could put in it, > but... > > Rich Brown <[email protected]> writes: > > > Update: (I thought I had sent the previous message yesterday. My mistake.) > > > > I now have atl3.richb-hanover.com running a netperf server. it's a > > stock Ubuntu 18.04.4 LTS - uname -a shows: Linux atl3 > > 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 > > x86_64 x86_64 GNU/Linux. I have installed netperf 2.6.0, and little > > else. > > I imagine fq_codel is the underlying qdisc? > > irtt would be nice. > > At some point, somehow, using some tool, we need to start supporting QUIC. > > > > > Next steps: > > > > 1) Please hammer on the server to see if it's a suitable replacement > > for the canonical "netperf.bufferbloat.net". Please feel free to check > > both its ability to handle traffic as well as any security surprises > > you discover... > > Thank you so much for maintaining this server for so long and I had no > idea before this thread that it was being so hard hit. > > > > 2) I welcome suggestions for configuring the server's TCP stack to be > > most useful for researchers. fq_codel, bbr, - I'm open to your > > thoughts. > > I have generally made available cubic, reno, bbr, and dctcp. I was > always rather reluctant to publish where I'd turned on dctcp given that > it only recently gained a response to packet loss. In fact, reluctant to > publish anything in the cloud. > > I had (briefly) an SCE capable machine up in singapore. It would be > good if more folk could try cubic-sce, reno-sce, dctcp-sce, bbr-sce, and > for that matter, play with l4s in the same dc on an equivalent machine, > but that involves making custom kernels for each at present. I'm also > dying to try out the ebpf + etx stuff google is presenting at netdevconf.... > > > > > 3) It's not too soon for advice on an iptables strategy for limiting > > the access/bandwidth/traffic to people who're abusing the service... > > I'd love to have that too! > > > Once we have all this in place, we can change the netperf.bufferbloat.net > > name to point to this server. Thanks. > > > > Rich > > > >> On Feb 8, 2020, at 5:35 PM, Rich Brown <[email protected]> wrote: > >> > >> Toke and Jesper, > >> > >> Thanks both for these responses. > >> > >> netperf.bufferbloat.net is running an OpenVZ VPS with a 3.10 > >> kernel. Tech support at Ramnode tells me that I need to get to a KVM > >> instance in order to use ipset and other fancy kernel stuff. > >> > >> Here's my plan: > >> > >> 1) Unless anyone can recommend a better hosting service ... > >> > >> 2) Over the weekend, I'll stand up a new KVM server at Ramnode. They > >> offer a 2GB RAM, 2 core, 65 GB SSD, with 3TB per month of > >> data. It'll cost $10/month: adding 2x1TB at $4/month brings it to a > >> total of $18/month, about what the current server costs. I can get > >> Ubuntu 18.04 LTS as a standard install. > >> > >> 3) While that's in-flight I would request that an iptables expert on > >> the list recommend a better strategy. (I was just makin' stuff up in > >> the current setup - as you could tell :-) > >> > >> 4) I'd also accept any thoughts about tc commands for setting up the > >> networking on the host to work best as a netperf server. (Maybe > >> enable fq_codel or better...) > >> > >> Thanks > >> > >> Rich > >> > >>> On Feb 7, 2020, at 7:02 AM, Jesper Dangaard Brouer <[email protected]> > >>> wrote: > >>> > >>> On Thu, 6 Feb 2020 18:47:06 -0500 > >>> Rich Brown <[email protected]> wrote: > >>> > >>>>> On Feb 6, 2020, at 12:00 PM, Matt Taggart wrote: > >>>>> > >>>>> This smells like a munin or smokeping plugin (or some other sort of > >>>>> monitoring) gathering data for graphing. > >>>> > >>>> Yup. That is a real possibility. The question is what we do about it. > >>>> > >>>> If I understood, we left it at: > >>>> > >>>> 1) Toke was going to look into some way to spread the > >>>> 'netperf.bufferbloat.net' load across several of our netperf servers. > >>>> > >>>> 2) Can someone give me advice about iptables/tc/? to identify IP > >>>> addresses that make "too many" connections and either shut them off > >>>> or dial their bandwidth back to a 3 or 5 kbps? > >>> > >>> Look at man iptables-extensions and find "connlimit" and "recent". > >>> > >>> > >>>> (If you're terminally curious, Line 5 of > >>>> https://github.com/richb-hanover/netperfclean/blob/master/addtoblacklist.sh > >>>> shows the current iptables command to drop connections from "heavy > >>>> users" identified in the findunfilteredips.sh script. You can read > >>>> the current iptables rules at: > >>>> https://github.com/richb-hanover/netperfclean/blob/master/iptables.txt) > >>> > >>> Sorry but this is a wrong approach. Creating an iptables rule per > >>> source IP-address, will (as you also demonstrate) give you a VERY long > >>> list of rules (which is evaluated sequentially by the kernel). > >>> > >>> This should instead be solved by using an ipset (howto a match from > >>> iptables see man iptables-extensions(8) and "set"). And use the > >>> cmdline tool ipset to add and remove entries. > >>> > >>> -- > >>> Best regards, > >>> Jesper Dangaard Brouer > >>> MSc.CS, Principal Kernel Engineer at Red Hat > >>> LinkedIn: http://www.linkedin.com/in/brouer > >>> > >> > > > > _______________________________________________ > > Bloat mailing list > > [email protected] > > https://lists.bufferbloat.net/listinfo/bloat > _______________________________________________ > Bloat mailing list > [email protected] > https://lists.bufferbloat.net/listinfo/bloat -- Make Music, Not War Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-435-0729 _______________________________________________ Bloat mailing list [email protected] https://lists.bufferbloat.net/listinfo/bloat
