Re: [Pdns-users] Acheiving 40,000 QPS
Hi Lee, In the past few months, quite some CNS servers have been replaced by the PowerDNS Recursor, so you are not alone! The 40,000 number has been reproduced by a number of separate parties, sample output is below. First, please read http://doc.powerdns.com/recursor-performance.html especially the bit about firewalls. PowerDNS does aggressive source port randomization, and may stress out intervening firewalls. Second, try: dnsperf -q 1000 -l 100 queryfile-example-3million I just did, and the output is: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 2995336 queries Queries completed:2991690 queries Queries lost: 3646 queries Avg request size: 41 bytes Avg response size:80 bytes Percentage completed: 99.88% Percentage lost:0.12% Started at: Mon May 17 08:23:12 2010 Finished at: Mon May 17 08:24:57 2010 Ran for: 104.828020 seconds Queries per second: 28539.029927 qps This is on a server with a firewall in place, and the first run. Second run with firewall state tracking disabled as per the instructions on doc.powerdns.com: Statistics: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 1 times Queries sent: 5700362 queries Queries completed:5694333 queries Queries lost: 6029 queries Avg request size: 41 bytes Avg response size:80 bytes Percentage completed: 99.89% Percentage lost:0.11% Started at: Mon May 17 08:26:33 2010 Finished at: Mon May 17 08:28:18 2010 Ran for: 104.71 seconds Queries per second: 54381.363598 qps I hope this helps answer your question. The main trick is to have enough queries operating in parallel, which incidentally is a close match to 'real life' heavy use operation. The operating system of the numbers above is Ubuntu Karmic Koala. Let me know if you have further questions. Kind regards, Bert Hubert On Mon, May 17, 2010 at 01:45:46PM +0800, Lee Standen wrote: Hi All, We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS. I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second. Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second? The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system. Thanks ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Acheiving 40,000 QPS
First things first: use threads=4 or threads=8 and packet-cache. This is how I did it: Run a couple of dnsperf's like this: ./dnsperf -d queryfile-example-3million -s IP -p port -t 2 -l $((3600*10)) -q 2000 -Q 2000 logfile.log 21 This will run a dnsperf from a queryfile for 10 hours (so you will see something on the graphs), with 2000qps. Now, run a couple of these simultaneously and see what happens. Probably not a scientific method for precise testing :), but you should see the qps on the graphs (make sure you create some graphs). You should run the dnsperfs from another machine IMHO (or maybe 2-3 other machines). Here's a bigger queryfile to replay if you need it (real life customer queries from two months ago): http://hosting.astral.ro/~gimre/queryfile-24mil.txt.bz2 Let us know how it goes. Disclaimer: I'm no (power)DNS expert or anything, so the above could be worthless :) On 05/17/2010 08:45 AM, Lee Standen wrote: Hi All, We’re looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we’re impressed by the performance claimed by PowerDNS. I’ve deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I’m only able to pull about 2500 requests per second. Does any documentation exist which describes which settings must be changed to achieve the claimed 40,000 requests per second? The server is currently deployed on an IBM HS-21XM blade, which is an 8 core Xeon system. Thanks ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users -- Imre Gergely Yahoo!: gergelyimre | ICQ#: 101510959 MSN: gergely_imre | GoogleTalk: gergelyimre gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Acheiving 40,000 QPS
Thanks for that information, Bert. I had seen the documentation before... perhaps I should provide some more information. Here are the options I'm using at the moment (I've tried several values): --- disable-packetcache=no local-address=0.0.0.0 local-port=53 max-cache-entries=400 max-packetcache-entries=400 single-socket=off threads=8 --- Here's the results of the second run using the same command line you had below... the data file I found using a quick google search for the filename you used: ./queryperf -q 1000 -l 100 queryfile-example-3million --- Statistics: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 1285008 queries Queries completed:1284175 queries Queries lost: 833 queries Queries delayed(?): 0 queries RTT max: 5.000198 sec RTT min: 0.02 sec RTT average: 0.074771 sec RTT std deviation:0.265890 sec RTT out of range: 1 queries Percentage completed: 99.94% Percentage lost:0.06% Started at: Mon May 17 07:15:32 2010 Finished at: Mon May 17 07:17:17 2010 Ran for: 104.875333 seconds Queries per second: 12244.776377 qps --- I just double-checked the hardware and I've actually got 4 cores available, not 8 as I initially thought: --- processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU5150 @ 2.66GHz stepping: 6 cpu MHz : 2667.294 cache size : 4096 KB --- Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04 rather than building a new binary. It was assumed that those binaries would be optimized, but if they're not, I'm happy to try building it myself. Thanks in advance for any suggestions you might have :) -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Monday, 17 May 2010 2:31 PM To: Lee Standen Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Acheiving 40,000 QPS Hi Lee, In the past few months, quite some CNS servers have been replaced by the PowerDNS Recursor, so you are not alone! The 40,000 number has been reproduced by a number of separate parties, sample output is below. First, please read http://doc.powerdns.com/recursor-performance.html especially the bit about firewalls. PowerDNS does aggressive source port randomization, and may stress out intervening firewalls. Second, try: dnsperf -q 1000 -l 100 queryfile-example-3million I just did, and the output is: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 2995336 queries Queries completed:2991690 queries Queries lost: 3646 queries Avg request size: 41 bytes Avg response size:80 bytes Percentage completed: 99.88% Percentage lost:0.12% Started at: Mon May 17 08:23:12 2010 Finished at: Mon May 17 08:24:57 2010 Ran for: 104.828020 seconds Queries per second: 28539.029927 qps This is on a server with a firewall in place, and the first run. Second run with firewall state tracking disabled as per the instructions on doc.powerdns.com: Statistics: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 1 times Queries sent: 5700362 queries Queries completed:5694333 queries Queries lost: 6029 queries Avg request size: 41 bytes Avg response size:80 bytes Percentage completed: 99.89% Percentage lost:0.11% Started at: Mon May 17 08:26:33 2010 Finished at: Mon May 17 08:28:18 2010 Ran for: 104.71 seconds Queries per second: 54381.363598 qps I hope this helps answer your question. The main trick is to have enough queries operating in parallel, which incidentally is a close match to 'real life' heavy use operation. The operating system of the numbers above is Ubuntu Karmic Koala. Let me know if you have further questions. Kind regards, Bert Hubert On Mon, May 17, 2010 at 01:45:46PM +0800, Lee Standen wrote: Hi All, We're looking to trial PowerDNS as an option to replace our Nominum CNS servers in the next few months, and we're impressed by the performance claimed by PowerDNS. I've deployed a recursive server and have attempted to replay some captured traffic, but based on my initial testing, I'm only able to pull about 2500 requests per second. Does any documentation exist
Re: [Pdns-users] Acheiving 40,000 QPS
Looks like I've narrowed down the issue from this setup. The issue was one of concurrency... by boosting it up to 4000 outstanding, I was able to break over the 30,000 per second mark. It's quite possible that some combination of settings means we need to run more concurrent queries to reach ./queryperf -q 4000 -l 60 queryfile-example-3million Given the hardware is below what I thought we were running on, I'm quite happy with the performance level -- further, the fact it only dropped 1.26% of queries when running at this performance level, I'm even more impressed. --- Statistics: Parse input file: multiple times Run time limit: 60 seconds Ran through file: 0 times Queries sent: 1982466 queries Queries completed:1957446 queries Queries lost: 25020 queries Queries delayed(?): 0 queries RTT max: 5.005290 sec RTT min: 0.03 sec RTT average: 0.061326 sec RTT std deviation:0.296004 sec RTT out of range: 10 queries Percentage completed: 98.74% Percentage lost:1.26% Started at: Mon May 17 07:48:36 2010 Finished at: Mon May 17 07:49:41 2010 Ran for: 65.16 seconds Queries per second: 30114.546433 qps -Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of Lee Standen Sent: Monday, 17 May 2010 3:23 PM To: bert hubert Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Acheiving 40,000 QPS Thanks for that information, Bert. I had seen the documentation before... perhaps I should provide some more information. Here are the options I'm using at the moment (I've tried several values): --- disable-packetcache=no local-address=0.0.0.0 local-port=53 max-cache-entries=400 max-packetcache-entries=400 single-socket=off threads=8 --- Here's the results of the second run using the same command line you had below... the data file I found using a quick google search for the filename you used: ./queryperf -q 1000 -l 100 queryfile-example-3million --- Statistics: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 1285008 queries Queries completed:1284175 queries Queries lost: 833 queries Queries delayed(?): 0 queries RTT max: 5.000198 sec RTT min: 0.02 sec RTT average: 0.074771 sec RTT std deviation:0.265890 sec RTT out of range: 1 queries Percentage completed: 99.94% Percentage lost:0.06% Started at: Mon May 17 07:15:32 2010 Finished at: Mon May 17 07:17:17 2010 Ran for: 104.875333 seconds Queries per second: 12244.776377 qps --- I just double-checked the hardware and I've actually got 4 cores available, not 8 as I initially thought: --- processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU5150 @ 2.66GHz stepping: 6 cpu MHz : 2667.294 cache size : 4096 KB --- Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04 rather than building a new binary. It was assumed that those binaries would be optimized, but if they're not, I'm happy to try building it myself. Thanks in advance for any suggestions you might have :) -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Monday, 17 May 2010 2:31 PM To: Lee Standen Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Acheiving 40,000 QPS Hi Lee, In the past few months, quite some CNS servers have been replaced by the PowerDNS Recursor, so you are not alone! The 40,000 number has been reproduced by a number of separate parties, sample output is below. First, please read http://doc.powerdns.com/recursor-performance.html especially the bit about firewalls. PowerDNS does aggressive source port randomization, and may stress out intervening firewalls. Second, try: dnsperf -q 1000 -l 100 queryfile-example-3million I just did, and the output is: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 2995336 queries Queries completed:2991690 queries Queries lost: 3646 queries Avg request size: 41 bytes Avg response size:80 bytes Percentage completed: 99.88% Percentage lost:0.12% Started at: Mon May 17 08:23:12 2010 Finished
Re: [Pdns-users] Acheiving 40,000 QPS
On Mon, May 17, 2010 at 03:23:21PM +0800, Lee Standen wrote: Thanks for that information, Bert. I had seen the documentation before... perhaps I should provide some more information. Here are the options I'm using at the moment (I've tried several values): --- disable-packetcache=no local-address=0.0.0.0 It might be wise to set this to the actual address you are using, but this will not impact performance. local-port=53 max-cache-entries=400 max-packetcache-entries=400 single-socket=off I'd remove this one. threads=8 4 might be better given your hardware. At 8, some scheduling infelicities might pop up. Percentage completed: 99.94% Percentage lost:0.06% Queries per second: 12244.776377 qps This is decidedly disappointing. Can you retry with threads=2 and threads=4? Can you try a 'second run' with the same data? This can rule out certain scnarios. I just double-checked the hardware and I've actually got 4 cores available, not 8 as I initially thought: Is this real hardware or virtualized? Lastly, I'm using the amd64 deb from the powerdns web site on Ubuntu 10.04 rather than building a new binary. It was assumed that those binaries would be optimized, but if they're not, I'm happy to try building it myself. They are pretty ok - the numbers I posted earlier are also from those debs. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, I didn't read the whole thread. But we had the same problem in past. Is it maybee a timeout mysql connection. We had this only system with out load. Ciao Marco Am 06.05.10 07:45, schrieb bert hubert: On Wed, May 05, 2010 at 11:27:13AM +0200, Ton van Rosmalen wrote: Our primary auth pdns didn't respond to tcp queries anymore where udp queries still worked. A restart solved the problem. I didn't think anything of it until I read these new messages on this thread. Could it be a shared problem between the recursor and auth server? Highly unlikely. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users - -- Marco Schrieck Bereichsleiter Entwicklung - -- InterNetX GmbH Maximilianstr. 6 93047 Regensburg Germany Tel. +49 941 59559-0 Fax +49 941 59579-050 www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Geschäftsführer/CEO: Thomas Mörz Amtsgericht Regensburg, HRB 7142 ICQ: 232016987 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJL8RUTAAoJEN9yMHEBd2HnGRAIAMBtTvOoXVRUgT9sl59uWCSR ZDfJJTotO0ZoIyI3PrksCK95nQKmb9n+KnYrz+ZYPI9F7cCv5tnqGHUvQ28IFoXW xzxnVm6T50n+68XRW2/kj6l139U08ebuJ6tzKC0gYLQW0IdbfayBKJbdRCnBOPDs AKprkh9fWqyFPdUb/AEOfOzCRX5rnBwKP/5jzm4J1xT00LKApfo2LaK4/GwQpWIH ujsj/fuP81Kz680IrU7STHroaiREFJRpzlk4ehDlqDr7lnLuu6VuZqnJcquTpxwV GNMK/3Dwhaa+JQupU+hS+Zm1AhZunOkv0iO/mJ15ETz1trrzM/xLAgKTzbH64/s= =6nTU -END PGP SIGNATURE- ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Acheiving 40,000 QPS
Perf testing of 2/4/6/8 threads still shows optimal performance at 4 threads. Cache size of 3M, using test file of 3M addresses. Tested on HP blades with quad-core X5570 CPU and 48GB RAM, 146GB 15k SAS drives running on CentOS 5.x with a custom RPM. 2 threads: new.2.1: Maximum throughput: 40316.00 qps new.2.2: Maximum throughput: 45260.00 qps new.2.3: Maximum throughput: 51250.00 qps new.2.4: Maximum throughput: 58752.00 qps new.2.5: Maximum throughput: 63724.00 qps 4 threads: new.4.1: Maximum throughput: 68176.00 qps new.4.2: Maximum throughput: 85418.00 qps new.4.3: Maximum throughput: 98502.00 qps new.4.4: Maximum throughput: 97442.00 qps new.4.5: Maximum throughput: 98750.00 qps 6 threads: new.6.1: Maximum throughput: 85418.00 qps new.6.2: Maximum throughput: 98660.00 qps new.6.3: Maximum throughput: 96212.00 qps new.6.4: Maximum throughput: 95466.00 qps new.6.5: Maximum throughput: 96830.00 qps 8 threads: new.8.1: Maximum throughput: 89584.00 qps new.8.2: Maximum throughput: 96630.00 qps new.8.3: Maximum throughput: 92082.00 qps new.8.4: Maximum throughput: 93842.00 qps new.8.5: Maximum throughput: 86078.00 qps FYI here is full output from the 'resperf' tool: DNS Resolution Performance Testing Tool Nominum Version 1.0.1.0 [Status] Sending [Status] Waiting for more responses [Status] Testing complete Statistics: Queries sent: 299 Queries completed:2989899 Queries lost: 10100 Ran for: 100.00 seconds Maximum throughput: 97442.00 qps Lost at that point: 1.32% Brad -Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of Lee Standen Sent: Monday, May 17, 2010 12:57 AM To: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Acheiving 40,000 QPS Looks like I've narrowed down the issue from this setup. The issue was one of concurrency... by boosting it up to 4000 outstanding, I was able to break over the 30,000 per second mark. It's quite possible that some combination of settings means we need to run more concurrent queries to reach ./queryperf -q 4000 -l 60 queryfile-example-3million Given the hardware is below what I thought we were running on, I'm quite happy with the performance level -- further, the fact it only dropped 1.26% of queries when running at this performance level, I'm even more impressed. --- Statistics: Parse input file: multiple times Run time limit: 60 seconds Ran through file: 0 times Queries sent: 1982466 queries Queries completed:1957446 queries Queries lost: 25020 queries Queries delayed(?): 0 queries RTT max: 5.005290 sec RTT min: 0.03 sec RTT average: 0.061326 sec RTT std deviation:0.296004 sec RTT out of range: 10 queries Percentage completed: 98.74% Percentage lost:1.26% Started at: Mon May 17 07:48:36 2010 Finished at: Mon May 17 07:49:41 2010 Ran for: 65.16 seconds Queries per second: 30114.546433 qps -Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of Lee Standen Sent: Monday, 17 May 2010 3:23 PM To: bert hubert Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Acheiving 40,000 QPS Thanks for that information, Bert. I had seen the documentation before... perhaps I should provide some more information. Here are the options I'm using at the moment (I've tried several values): --- disable-packetcache=no local-address=0.0.0.0 local-port=53 max-cache-entries=400 max-packetcache-entries=400 single-socket=off threads=8 --- Here's the results of the second run using the same command line you had below... the data file I found using a quick google search for the filename you used: ./queryperf -q 1000 -l 100 queryfile-example-3million --- Statistics: Parse input file: multiple times Run time limit: 100 seconds Ran through file: 0 times Queries sent: 1285008 queries Queries completed:1284175 queries Queries lost: 833 queries Queries delayed(?): 0 queries RTT max: 5.000198 sec RTT min: 0.02 sec RTT average: 0.074771 sec RTT std deviation:0.265890 sec RTT out of range: 1 queries Percentage completed: 99.94% Percentage lost:0.06% Started at: Mon May 17 07:15:32 2010 Finished at: Mon May 17 07:17:17 2010 Ran for: 104.875333 seconds Queries per second:
