Re: Slow downloads through mod_jk on Windows XP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy, On 5/10/12 6:36 PM, Andy Wang wrote: Direct to apache http: Transfer rate: 21925.90 [Kbytes/sec] received Through AJP: Transfer rate: 36732.95 [Kbytes/sec] received Direct to tomcat http: Transfer rate: 30968.31 [Kbytes/sec] received Am I reading this right: httpd+AJP outperforms both direct-to-httpd and direct-to-Tomcat? That seems like one of the following are true: 1. The laws of physics don't behave as we expect them to in this case, and you should donate your rig to CERN. 2. Your data is so sensitive to some other external conditions that this is a meaningless test. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+xKLIACgkQ9CaO5/Lv0PDBiACffaCipEkfYg2kEQrPEpeyvUfY dQEAn0xoRq9LCdUVcI4FXBEEbHcWXpIq =B/8N -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
On 05/11/2012 04:51 AM, Mladen Turk wrote: I was following this tread and was hoping that someone will say: Do not use workstation grade software for server applications but no. XP (Win7 falls in the same category) has good network stack but focused on client applications. There is a good reason why MS charges extra $$ for server versions since they make sure your server application don't perform well on workstation software. The problem is, we've seen the behaviour reported sporadically from customers on windows 2003 server. The only case I've been able to find to reproduce it was this XP system. I'm not sure what to make of the customer reports of the slow downs but at this point, I'm going to have to ask them to use something like ab.exe to do the downloads instead of Internet Explorer (most of them use IE to do it). Maybe there's some stupidity with IE. IE tries to open several concurrent connection at once (RFC says that number is max 3) I'm sure you can simulate that with 'ab -c 10 -k -n 1 ...' I'm not sure how this would be relevant though as the behavior is reproducible with a single connection to download a large file (no concurrenct connections to this site or any other site. There is nothing wrong with tomcat, mod_jk or httpd. It's just your use case. I'm not convinced that there isn't something odd going on but for now I'll ignore it. I may have a stupid use case, but we have had reports out in the field of what appear to be perfectly sane use cases that don't perform. i just can't reproduce them. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
On 10/05/2012 23:36, Andy Wang wrote: So I cannot reproduce the slow down to 4-5MB/s on the same VM I was able to reproduce it on once I copied the VM to an adequate vmware server. But I do see some neat numbers in case people care. I ran with ab -5 directly against apache, against a url mapped to ajp as well as direct to the http connectors. The numbers were consistent between tomcat 5.0, 5.5, 6.0 and 7.0 so I'm only going to post one set of numbers for tomcat. This is against a windows XP SP3 with no sendbuffersize, tcpbuffersize or scaling window tuning: Direct to apache http: Transfer rate: 21925.90 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.3 1 1 Processing: 19593 20077 474.0 20045 20855 Waiting:12 0.6 2 3 Total: 19593 20078 474.1 20046 20856 Through AJP: Transfer rate: 36732.95 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10662 11984 879.6 12227 12975 Waiting:45 0.5 5 5 Total: 10663 11984 879.7 12227 12976 Direct to tomcat http: Transfer rate: 30968.31 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 1.6 0 4 Processing: 11326 14214 2655.1 15565 16952 Waiting:35 1.6 5 7 Total: 11326 14215 2654.3 15565 16952 Note how much better the both the tomcat results are than direct apache. Most interestingly, note how much better AJP is than direct tomcat HTTP connector. That was quite unexpected. Here are the results from a Windows 2008 system on the same vm host: Direct to apache http: Transfer rate: 57968.69 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.1 1 1 Processing: 7453 7594 181.8 75757890 Waiting:2 12 18.5 6 45 Total: 7453 7594 181.8 75767890 Through AJP: Transfer rate: 31532.82 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10723 13960 2813.3 15795 16409 Waiting:35 3.1 4 10 Total: 10724 13961 2813.4 15795 16410 Direct to Tomcat http: Transfer rate: 37742.45 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 0.1 0 1 Processing: 10974 11664 452.1 11812 12192 Waiting:2 14 25.2 4 59 Total: 10974 11664 452.2 11813 12192 Tomcat http averaged better times, BUT ajp was able to perform faster at times. Direct HTTP to apache is way faster though but I think that's to be expected. I'd guess that the default Context cache settings are giving Tomcat an edge here. HTTPD does not have caching enable OOTB. p So realistically, I think the 2008 numbers make sense to me and the XP numbers show that XPs tcp stack is a piece of crap (which I think alot of people already know). I'm not sure what to make of the customer reports of the slow downs but at this point, I'm going to have to ask them to use something like ab.exe to do the downloads instead of Internet Explorer (most of them use IE to do it). Maybe there's some stupidity with IE. Anyways, I'm closing the book on this (with a bookmark just in case) but wanted to provide the numbers in case people were curious what I got. Thanks, Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Slow downloads through mod_jk on Windows XP
On 05/11/2012 12:36 AM, Andy Wang wrote: So I cannot reproduce the slow down to 4-5MB/s on the same VM I was able to reproduce it on once I copied the VM to an adequate vmware server. But I do see some neat numbers in case people care. I was following this tread and was hoping that someone will say: Do not use workstation grade software for server applications but no. XP (Win7 falls in the same category) has good network stack but focused on client applications. There is a good reason why MS charges extra $$ for server versions since they make sure your server application don't perform well on workstation software. I'm not sure what to make of the customer reports of the slow downs but at this point, I'm going to have to ask them to use something like ab.exe to do the downloads instead of Internet Explorer (most of them use IE to do it). Maybe there's some stupidity with IE. IE tries to open several concurrent connection at once (RFC says that number is max 3) I'm sure you can simulate that with 'ab -c 10 -k -n 1 ...' Anyways, I'm closing the book on this (with a bookmark just in case) but wanted to provide the numbers in case people were curious what I got. There is nothing wrong with tomcat, mod_jk or httpd. It's just your use case. Regards -- ^TM - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow downloads through mod_jk on Windows XP
Andy- On the receive side *if you have windows scaling enabled* you supposedly *can* enhance the receive window by setting these registry keys HKEY_LOCAL_MACHINE\System \CurrentControlSet\Services\Tcpip\Parameters\TCPWindowSize HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\Interface\TCPWindowSize registry values can be set to a maximum of 65,535 bytes (without window scaling) or 1,073,741,823 (with window scaling). Microsoft does admit to 5GBPS hard limit for receive On the send side you can increase thoughput by enabling CTCP (Compound TCP) netsh interface tcp set global congestionprovider=ctcp Loss negotiation can be mitigated by enabling Enhanced Control Notification algoithm via netsh interface tcp set global ecncapability=enabled http://technet.microsoft.com/library/bb878127 I remember QOS agents installed on both send and receive sides helped monitor and enhance Throughput Do you have QOS agents you can enable to track and reconfig send and receive parameters? Thanks for the hard work collecting these stats..this will help me determine which connector to use going forward Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. Date: Thu, 10 May 2012 17:36:02 -0500 From: aw...@ptc.com To: users@tomcat.apache.org Subject: Re: Slow downloads through mod_jk on Windows XP So I cannot reproduce the slow down to 4-5MB/s on the same VM I was able to reproduce it on once I copied the VM to an adequate vmware server. But I do see some neat numbers in case people care. I ran with ab -5 directly against apache, against a url mapped to ajp as well as direct to the http connectors. The numbers were consistent between tomcat 5.0, 5.5, 6.0 and 7.0 so I'm only going to post one set of numbers for tomcat. This is against a windows XP SP3 with no sendbuffersize, tcpbuffersize or scaling window tuning: Direct to apache http: Transfer rate: 21925.90 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.3 1 1 Processing: 19593 20077 474.0 20045 20855 Waiting:12 0.6 2 3 Total: 19593 20078 474.1 20046 20856 Through AJP: Transfer rate: 36732.95 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10662 11984 879.6 12227 12975 Waiting:45 0.5 5 5 Total: 10663 11984 879.7 12227 12976 Direct to tomcat http: Transfer rate: 30968.31 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 1.6 0 4 Processing: 11326 14214 2655.1 15565 16952 Waiting:35 1.6 5 7 Total: 11326 14215 2654.3 15565 16952 Note how much better the both the tomcat results are than direct apache. Most interestingly, note how much better AJP is than direct tomcat HTTP connector. That was quite unexpected. Here are the results from a Windows 2008 system on the same vm host: Direct to apache http: Transfer rate: 57968.69 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.1 1 1 Processing: 7453 7594 181.8 75757890 Waiting:2 12 18.5 6 45 Total: 7453 7594 181.8 75767890 Through AJP: Transfer rate: 31532.82 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10723 13960 2813.3 15795 16409 Waiting:35 3.1 4 10 Total: 10724 13961 2813.4 15795 16410 Direct to Tomcat http: Transfer rate: 37742.45 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 0.1
Re: Slow downloads through mod_jk on Windows XP
On 1:59 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy, On 5/8/12 6:41 PM, Andy Wang wrote: Initial benchmarks seem to show that the behavior between tomcats is not an issue. Do you mean that Tomcat performance appears to be the same regardless of version? That's both good and bad... I thought there were some performance improvements to the connectors from 5.5- 6.0. Maybe that was 4.x-5.5. Tomcat7 is using JDK 1.7 and this is interesting. The benchmarks with tomcat7+jdk1.7 vary widely across the board (both through ajp and direct http to tomcat) from 30s-40sMB/s. Java 1.6 seems alot more consistent. Not sure why yet. That is interesting. On the other hand, the server /is/ on a virtual machine, and you never know what other processes are stealing focus. Many VMs are notorious for bad IO throughput (I'm looking at you, OpenVZ). I've also moved off the crappy Windows XP VM I was provided to a more recent Windows 2008 VM as well as a fresh Windows XP SP3 VM. In past experience it seems windows XP and windows 2003 were the worst of the bunch with the ajp downloads dropping as low as 4-5MB/s over time. Have you tried bare hardware? How much control do you have over the VM server? Seems to me that resources/performance could be not only affected by other VMs but also intentionally throttled. -Terence Bandoian I'm going to run a barrage of tests and provide the numbers. Do you think ab -n 5 and allowing ab to average the values of 5 hits for the ~440MB iso is a sound average? Some tips for this kind of testing: 1. Don't run ab on localhost: all the numbers will be worthless 2. Run ab with a range of concurrencies, including c=1 3. Make /lots/ of requests. IMO, 5 requests is really a pinhole analysis. I would make as many requests as you can over 10 minutes and see what the throughput ends up being. I'll compare Windows XP performance and Windows 2008 performance and after that I'll do the same on a Linux VM to get a better comparison. It will be good to see. If you want some really crappy scripts to get you going, feel free to start with mine from a while back (look in the scripts subdir): http://people.apache.org/~schultz/ApacheCon%20NA%202010/ Those scripts can run a ... lot of ab tests with lots of different concurrencies against a series of URLs -- that allows you to set up everything with, say, a different path or port number to get the various setups (bare httpd, httpd+mod_jk, httpd+mod_proxy, etc.) and then let it run all night. It will also produce some tables for you that can then easily be graphed. I also did bump up the ajpPacket size to 64K with no noticeable change to the benchmark numbers. So while 8k seems crappy it doesn't seem to be an issue. Given that apache and tomcat are both local I wouldn't expect that to be a big problem with 8k chunks given the near non-existent latency of local connections. It's good to know that the packet size didn't affect performance, but I agree that localhost communication is always magically-fast no mater what. I plan on doing both local ab requests as well as remote. The problem with remote is that our network is busy, so it may account for some variations but I don't think I can get our IT to segment me anything for this purpose :(. Just get a crossover cable and use static IP addresses. I'm not so concerned about a 25% hit. I'm really more concerned with the drop to 4-5MB/s over time that seems to happen. Does this happen locally or only remotely? I wonder if you're hitting some kind of traffic-shaping or QOS rules on your own internal network. - -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
I have solid numbers that I will e-mail in a follow up by itself so it's not lossed in the shuffle. Some answers to the comments inline. Thanks, Andy Do you mean that Tomcat performance appears to be the same regardless of version? That's both good and bad... I thought there were some performance improvements to the connectors from 5.5- 6.0. Maybe that was 4.x-5.5. Yeah, the download performance through AJP and through HTTP connectors is fairly consistent from 5.5-6.0-7.0 Tomcat7 is using JDK 1.7 and this is interesting. The benchmarks with tomcat7+jdk1.7 vary widely across the board (both through ajp and direct http to tomcat) from 30s-40sMB/s. Java 1.6 seems alot more consistent. Not sure why yet. That is interesting. On the other hand, the server /is/ on a virtual machine, and you never know what other processes are stealing focus. Many VMs are notorious for bad IO throughput (I'm looking at you, OpenVZ). I'm using vmware 5 esxi servers on a pretty beefy system. 8 2.8ghz xeon cores, 32gb memory and I'm only running a single VM at a time. Each VM is allocated 4GB of memory and 2 cores. Should be more than enough to handle serving up a 450MB iso file. Have you tried bare hardware? No, unfortunately I don't have access to bare hardware, but as I mentioned, the vmware 5 esxi servers are pretty beefy. How much control do you have over the VM server? Seems to me that resources/performance could be not only affected by other VMs but also intentionally throttled. I have pretty much full control over the VM servers. There is no throttling going on, no resource limits and as I said, I shut down all the other VMs just to rule out contention with them. Some tips for this kind of testing: 1. Don't run ab on localhost: all the numbers will be worthless 2. Run ab with a range of concurrencies, including c=1 3. Make /lots/ of requests. IMO, 5 requests is really a pinhole analysis. I would make as many requests as you can over 10 minutes and see what the throughput ends up being. I actually ran localhost just out of curiosity and while the numbers are faster, the performance difference is consistent with a remote host. I don't want to make alot of requests. I want to see throughput of large files becuase this is where we're seeing the slow downs. Concurrency is not an issue. I'll compare Windows XP performance and Windows 2008 performance and after that I'll do the same on a Linux VM to get a better comparison. It will be good to see. I decided to just do windows. The numbers I got are enough for me to not care at this point. I plan on doing both local ab requests as well as remote. The problem with remote is that our network is busy, so it may account for some variations but I don't think I can get our IT to segment me anything for this purpose :(. Just get a crossover cable and use static IP addresses. Unfortunately no access to additional hardware. I'm not so concerned about a 25% hit. I'm really more concerned with the drop to 4-5MB/s over time that seems to happen. Does this happen locally or only remotely? I wonder if you're hitting some kind of traffic-shaping or QOS rules on your own internal network. This type of hit occurs for our customers both locally and remotely. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
So I cannot reproduce the slow down to 4-5MB/s on the same VM I was able to reproduce it on once I copied the VM to an adequate vmware server. But I do see some neat numbers in case people care. I ran with ab -5 directly against apache, against a url mapped to ajp as well as direct to the http connectors. The numbers were consistent between tomcat 5.0, 5.5, 6.0 and 7.0 so I'm only going to post one set of numbers for tomcat. This is against a windows XP SP3 with no sendbuffersize, tcpbuffersize or scaling window tuning: Direct to apache http: Transfer rate: 21925.90 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.3 1 1 Processing: 19593 20077 474.0 20045 20855 Waiting:12 0.6 2 3 Total: 19593 20078 474.1 20046 20856 Through AJP: Transfer rate: 36732.95 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10662 11984 879.6 12227 12975 Waiting:45 0.5 5 5 Total: 10663 11984 879.7 12227 12976 Direct to tomcat http: Transfer rate: 30968.31 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 1.6 0 4 Processing: 11326 14214 2655.1 15565 16952 Waiting:35 1.6 5 7 Total: 11326 14215 2654.3 15565 16952 Note how much better the both the tomcat results are than direct apache. Most interestingly, note how much better AJP is than direct tomcat HTTP connector. That was quite unexpected. Here are the results from a Windows 2008 system on the same vm host: Direct to apache http: Transfer rate: 57968.69 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.1 1 1 Processing: 7453 7594 181.8 75757890 Waiting:2 12 18.5 6 45 Total: 7453 7594 181.8 75767890 Through AJP: Transfer rate: 31532.82 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 0.2 1 1 Processing: 10723 13960 2813.3 15795 16409 Waiting:35 3.1 4 10 Total: 10724 13961 2813.4 15795 16410 Direct to Tomcat http: Transfer rate: 37742.45 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 0.1 0 1 Processing: 10974 11664 452.1 11812 12192 Waiting:2 14 25.2 4 59 Total: 10974 11664 452.2 11813 12192 Tomcat http averaged better times, BUT ajp was able to perform faster at times. Direct HTTP to apache is way faster though but I think that's to be expected. So realistically, I think the 2008 numbers make sense to me and the XP numbers show that XPs tcp stack is a piece of crap (which I think alot of people already know). I'm not sure what to make of the customer reports of the slow downs but at this point, I'm going to have to ask them to use something like ab.exe to do the downloads instead of Internet Explorer (most of them use IE to do it). Maybe there's some stupidity with IE. Anyways, I'm closing the book on this (with a bookmark just in case) but wanted to provide the numbers in case people were curious what I got. Thanks, Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
On 08/05/2012 22:13, Caldarale, Charles R wrote: From: Andy Wang [mailto:aw...@ptc.com] Subject: Re: Slow downloads through mod_jk on Windows XP Downloading a large file through mod_jk to tomcat looks like this: 2012-05-08 16:01:22 (15.0 MB/s) - sol-11--text-x86.iso.8 saved [450799616/450799616] Downloading the same large file directly through apache looks like: 2012-05-08 16:01:58 (19.3 MB/s) - sol-11--text-x86.iso.11 saved [450799616/450799616] So apache still beats tomcat by a good chunk No, httpd (not Apache, which is an open source software organization) beats httpd+Tomcat - any other result would be violating several laws of physics. Might be interesting to measure just Tomcat in your environment. But preferably a supported non-ancient version of Tomcat. p - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Slow downloads through mod_jk on Windows XP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy, On 5/8/12 6:41 PM, Andy Wang wrote: Initial benchmarks seem to show that the behavior between tomcats is not an issue. Do you mean that Tomcat performance appears to be the same regardless of version? That's both good and bad... I thought there were some performance improvements to the connectors from 5.5- 6.0. Maybe that was 4.x-5.5. Tomcat7 is using JDK 1.7 and this is interesting. The benchmarks with tomcat7+jdk1.7 vary widely across the board (both through ajp and direct http to tomcat) from 30s-40sMB/s. Java 1.6 seems alot more consistent. Not sure why yet. That is interesting. On the other hand, the server /is/ on a virtual machine, and you never know what other processes are stealing focus. Many VMs are notorious for bad IO throughput (I'm looking at you, OpenVZ). I've also moved off the crappy Windows XP VM I was provided to a more recent Windows 2008 VM as well as a fresh Windows XP SP3 VM. In past experience it seems windows XP and windows 2003 were the worst of the bunch with the ajp downloads dropping as low as 4-5MB/s over time. Have you tried bare hardware? I'm going to run a barrage of tests and provide the numbers. Do you think ab -n 5 and allowing ab to average the values of 5 hits for the ~440MB iso is a sound average? Some tips for this kind of testing: 1. Don't run ab on localhost: all the numbers will be worthless 2. Run ab with a range of concurrencies, including c=1 3. Make /lots/ of requests. IMO, 5 requests is really a pinhole analysis. I would make as many requests as you can over 10 minutes and see what the throughput ends up being. I'll compare Windows XP performance and Windows 2008 performance and after that I'll do the same on a Linux VM to get a better comparison. It will be good to see. If you want some really crappy scripts to get you going, feel free to start with mine from a while back (look in the scripts subdir): http://people.apache.org/~schultz/ApacheCon%20NA%202010/ Those scripts can run a ... lot of ab tests with lots of different concurrencies against a series of URLs -- that allows you to set up everything with, say, a different path or port number to get the various setups (bare httpd, httpd+mod_jk, httpd+mod_proxy, etc.) and then let it run all night. It will also produce some tables for you that can then easily be graphed. I also did bump up the ajpPacket size to 64K with no noticeable change to the benchmark numbers. So while 8k seems crappy it doesn't seem to be an issue. Given that apache and tomcat are both local I wouldn't expect that to be a big problem with 8k chunks given the near non-existent latency of local connections. It's good to know that the packet size didn't affect performance, but I agree that localhost communication is always magically-fast no mater what. I plan on doing both local ab requests as well as remote. The problem with remote is that our network is busy, so it may account for some variations but I don't think I can get our IT to segment me anything for this purpose :(. Just get a crossover cable and use static IP addresses. I'm not so concerned about a 25% hit. I'm really more concerned with the drop to 4-5MB/s over time that seems to happen. Does this happen locally or only remotely? I wonder if you're hitting some kind of traffic-shaping or QOS rules on your own internal network. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+qxKYACgkQ9CaO5/Lv0PAs4QCgtAhZuB0Xc9MJJ0yo1YJPE3VQ P9UAoKRs+qE/DsdonYhCYI+WUkujCCoK =GVQX -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
On 05/07/2012 06:50 PM, Andy Wang wrote: On 05/07/2012 06:06 PM, André Warnier wrote: Considering your setup, it should not be too hard to set up a download of the same file file directly from Tomcat (through its HTTP Connector), to compare that with your two previous ways. This way, you could make sure if it is Tomcat, or the mod_jk/AJP link which is the issue. Also, still considering your setup, it should be possible to configure things so that these file downloads are handled directly by Apache httpd, since that seems to satisfy your expectations. mod_jk JkMount/JkUnMount rules (*) should make that possible, no ? Have to be a bit careful not to introduce security holes, and I am assuming that the files are static (which may be wrong here). (*) or the Location .. + setHandler jakarta-servlet configuration variation Thanks for the http connector idea. I forgot about that. The primary reason why i'm using tomcat to download a static file is really for testing purposes to confirm performance between mod_jk and direct apache. we have servlets that stream content files that see the same massive performance hit so in our actual use case it's not static files :(. I'm thinking this would be a valid test to help at least tweak mod_jk to it's potential. We've checked and double checked the buffering code of the servlets and it all looks fine AND the performance is fine on Linux and the speed characteristics are identical to serving static files through tomcat + mod_jk so I'm hoping that it's an apples to apples comparison. Andy Through the HTTP connector the performance is similar to apache direct. 30mb/s So there's something interesting going on specifically ajp. Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy, On 5/7/12 6:38 PM, Andy Wang wrote: This is Apache 2.2.22 and mod_jk 1.2.32 with tomcat 5.0.30 (yeah, I know this is ancient. I'll try with something newer tomorrow). Which connector are you using? If you have APR available, AJP should use the APR connector by default. Do you know if you are using APR/native? If not, consider trying it, and use sendFile=true. I'm not sure if it will improve anything because the real problem might be the actual buffering between Tomcat and httpd. Unfortunately, this is windows, and both apache and tomcat are local on the server so I can't sniff the packets between the two systems with wireshark. Wireshark should allow you to sniff traffic on localhost. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+pcmIACgkQ9CaO5/Lv0PCS1gCeI7k+2Mvg9O2uFUEfxgxhbtrU vH8An2js/6x0MubgspXUlTASJNRjiWpS =DOGG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
Which connector are you using? If you have APR available, AJP should use the APR connector by default. Do you know if you are using APR/native? If not, consider trying it, and use sendFile=true. I'm not sure if it will improve anything because the real problem might be the actual buffering between Tomcat and httpd. Unfortunately, we found some odd problems with APR/native a long time ago and haven't ever gone back to it. I don't have it available for the systems I need this working on to try Wireshark should allow you to sniff traffic on localhost. http://wiki.wireshark.org/CaptureSetup/Loopback It doesn't work on Windows. I've tried the loopback adapter piece, but it's quite obnoxious. What am I saying, Windows is quite obnoxious :) Andy
Re: Slow downloads through mod_jk on Windows XP
After playing with this a bit more (testing this time against tomcat 7.0.27) the ajpPacketSize has zero effect on the speed. Downloading a large file through mod_jk to tomcat looks like this: 2012-05-08 16:01:22 (15.0 MB/s) - “sol-11--text-x86.iso.8” saved [450799616/450799616] Downloading the same large file directly through apache looks like: 2012-05-08 16:01:58 (19.3 MB/s) - “sol-11--text-x86.iso.11” saved [450799616/450799616] the numbers are pretty consistent. So apache still beats tomcat by a good chunk but it's much much better than with tomcat 5.0 where the through tomcat numbers were about half that. I'm still wondering if there's something that can be tweaked in the MS TCP/IP stack to bring the two together closer. Andy On 05/08/2012 02:06 PM, Andy Wang wrote: On 05/07/2012 06:50 PM, Andy Wang wrote: On 05/07/2012 06:06 PM, André Warnier wrote: Considering your setup, it should not be too hard to set up a download of the same file file directly from Tomcat (through its HTTP Connector), to compare that with your two previous ways. This way, you could make sure if it is Tomcat, or the mod_jk/AJP link which is the issue. Also, still considering your setup, it should be possible to configure things so that these file downloads are handled directly by Apache httpd, since that seems to satisfy your expectations. mod_jk JkMount/JkUnMount rules (*) should make that possible, no ? Have to be a bit careful not to introduce security holes, and I am assuming that the files are static (which may be wrong here). (*) or the Location .. + setHandler jakarta-servlet configuration variation Thanks for the http connector idea. I forgot about that. The primary reason why i'm using tomcat to download a static file is really for testing purposes to confirm performance between mod_jk and direct apache. we have servlets that stream content files that see the same massive performance hit so in our actual use case it's not static files :(. I'm thinking this would be a valid test to help at least tweak mod_jk to it's potential. We've checked and double checked the buffering code of the servlets and it all looks fine AND the performance is fine on Linux and the speed characteristics are identical to serving static files through tomcat + mod_jk so I'm hoping that it's an apples to apples comparison. Andy Through the HTTP connector the performance is similar to apache direct. 30mb/s So there's something interesting going on specifically ajp. Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow downloads through mod_jk on Windows XP
From: Andy Wang [mailto:aw...@ptc.com] Subject: Re: Slow downloads through mod_jk on Windows XP Downloading a large file through mod_jk to tomcat looks like this: 2012-05-08 16:01:22 (15.0 MB/s) - sol-11--text-x86.iso.8 saved [450799616/450799616] Downloading the same large file directly through apache looks like: 2012-05-08 16:01:58 (19.3 MB/s) - sol-11--text-x86.iso.11 saved [450799616/450799616] So apache still beats tomcat by a good chunk No, httpd (not Apache, which is an open source software organization) beats httpd+Tomcat - any other result would be violating several laws of physics. Might be interesting to measure just Tomcat in your environment. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
Caldarale, Charles R wrote: From: Andy Wang [mailto:aw...@ptc.com] Subject: Re: Slow downloads through mod_jk on Windows XP Downloading a large file through mod_jk to tomcat looks like this: 2012-05-08 16:01:22 (15.0 MB/s) - sol-11--text-x86.iso.8 saved [450799616/450799616] Downloading the same large file directly through apache looks like: 2012-05-08 16:01:58 (19.3 MB/s) - sol-11--text-x86.iso.11 saved [450799616/450799616] So apache still beats tomcat by a good chunk No, httpd (not Apache, which is an open source software organization) beats httpd+Tomcat - any other result would be violating several laws of physics. Might be interesting to measure just Tomcat in your environment. He did that previously, and the result seemed to be that Tomcat alone was comparable to httpd alone, and both were better than httpd/mod_jk + Tomcat; which is indeed physically to be expected : more tubing, less throughput (excepting quantum tunelling effects of course). The question is more : how much of a degradation ? 15.0/19.3 = 0.77 = 23% less throughput. I don't know if this is a good chunk or the to be expected kind of degradation. According to the (looking seriously outdated) AJP protocol documentation at http://tomcat.apache.org/connectors-doc/ajp/ajpv13a.html, it would seem that the maximum data size chunk which AJP can send back from Tomcat to the front-end httpd is 8K at a time. So AJP might not be very well suited, when it comes to send back big blobs of data. Rainer would need to confirm if that is still the case now. One earlier message seemed to indicate that this httpd/mod_jk+tomcat deficit only happened under Windows though, and not under Linux. If that is confirmed, maybe there is some subtle difference in how the TCP/IP stack is being used under the one vs the other ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
He did that previously, and the result seemed to be that Tomcat alone was comparable to httpd alone, and both were better than httpd/mod_jk + Tomcat; which is indeed physically to be expected : more tubing, less throughput (excepting quantum tunelling effects of course). The question is more : how much of a degradation ? 15.0/19.3 = 0.77 = 23% less throughput. I don't know if this is a good chunk or the to be expected kind of degradation. According to the (looking seriously outdated) AJP protocol documentation at http://tomcat.apache.org/connectors-doc/ajp/ajpv13a.html, it would seem that the maximum data size chunk which AJP can send back from Tomcat to the front-end httpd is 8K at a time. So AJP might not be very well suited, when it comes to send back big blobs of data. Rainer would need to confirm if that is still the case now. One earlier message seemed to indicate that this httpd/mod_jk+tomcat deficit only happened under Windows though, and not under Linux. If that is confirmed, maybe there is some subtle difference in how the TCP/IP stack is being used under the one vs the other ? Thanks for that summary, That's about what I'm seeing. I just created a directory containing Apache configured to serve up the iso file directly as well as through three tomcats (tomcat5.5, tomcat6, and tomcat7) to see if the behavior is related to tomcat that I can easily copy between different Windows systems. Initial benchmarks seem to show that the behavior between tomcats is not an issue.Tomcat7 is using JDK 1.7 and this is interesting. The benchmarks with tomcat7+jdk1.7 vary widely across the board (both through ajp and direct http to tomcat) from 30s-40sMB/s. Java 1.6 seems alot more consistent. Not sure why yet. I've also moved off the crappy Windows XP VM I was provided to a more recent Windows 2008 VM as well as a fresh Windows XP SP3 VM. In past experience it seems windows XP and windows 2003 were the worst of the bunch with the ajp downloads dropping as low as 4-5MB/s over time. I'm going to run a barrage of tests and provide the numbers. Do you think ab -n 5 and allowing ab to average the values of 5 hits for the ~440MB iso is a sound average? I'll compare Windows XP performance and Windows 2008 performance and after that I'll do the same on a Linux VM to get a better comparison. I also did bump up the ajpPacket size to 64K with no noticeable change to the benchmark numbers. So while 8k seems crappy it doesn't seem to be an issue. Given that apache and tomcat are both local I wouldn't expect that to be a big problem with 8k chunks given the near non-existent latency of local connections. I plan on doing both local ab requests as well as remote. The problem with remote is that our network is busy, so it may account for some variations but I don't think I can get our IT to segment me anything for this purpose :(. I'm not so concerned about a 25% hit. I'm really more concerned with the drop to 4-5MB/s over time that seems to happen. Thanks, Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Slow downloads through mod_jk on Windows XP
Hi all, We've had a number of cases of people reporting to us that file downloads are slow when passed through tomcat and I've not been able to reproduce the problem on Linux but finally was provided a windows XP VM that was able to reproduce the problem. This is Apache 2.2.22 and mod_jk 1.2.32 with tomcat 5.0.30 (yeah, I know this is ancient. I'll try with something newer tomorrow). I have two URLs configured both with an identical iso file (roughly 450MB) in size. When I go through the URL configured to run directly through Apache the download speeds on the file are around 30MB/s which is reasonable for what I would expect on the network connection between the client and the web server. The second URL is JkMount'ed through to tomcat and download the same file drops down to 5-6 MB/s (starts at 10-ish MB/s). The one thing I haven't tried configuring is upping the ajp packet size because the version of tomcat I was provided doesn't support that, but otherwise the operating system is tuned so the TcpWindowSize is 131072. Unfortunately, this is windows, and both apache and tomcat are local on the server so I can't sniff the packets between the two systems with wireshark. I don't see anything on the mod_jk workers.properties configuration that deals with buffer sizes. Hoping someone has some ideas on what to look at that might help tweak this thing to perform similarly to Apache direct downloads. Thanks, Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
Andy Wang wrote: Hi all, We've had a number of cases of people reporting to us that file downloads are slow when passed through tomcat and I've not been able to reproduce the problem on Linux but finally was provided a windows XP VM that was able to reproduce the problem. This is Apache 2.2.22 and mod_jk 1.2.32 with tomcat 5.0.30 (yeah, I know this is ancient. I'll try with something newer tomorrow). I have two URLs configured both with an identical iso file (roughly 450MB) in size. When I go through the URL configured to run directly through Apache the download speeds on the file are around 30MB/s which is reasonable for what I would expect on the network connection between the client and the web server. The second URL is JkMount'ed through to tomcat and download the same file drops down to 5-6 MB/s (starts at 10-ish MB/s). The one thing I haven't tried configuring is upping the ajp packet size because the version of tomcat I was provided doesn't support that, but otherwise the operating system is tuned so the TcpWindowSize is 131072. Unfortunately, this is windows, and both apache and tomcat are local on the server so I can't sniff the packets between the two systems with wireshark. I don't see anything on the mod_jk workers.properties configuration that deals with buffer sizes. Hoping someone has some ideas on what to look at that might help tweak this thing to perform similarly to Apache direct downloads. Considering your setup, it should not be too hard to set up a download of the same file file directly from Tomcat (through its HTTP Connector), to compare that with your two previous ways. This way, you could make sure if it is Tomcat, or the mod_jk/AJP link which is the issue. Also, still considering your setup, it should be possible to configure things so that these file downloads are handled directly by Apache httpd, since that seems to satisfy your expectations. mod_jk JkMount/JkUnMount rules (*) should make that possible, no ? Have to be a bit careful not to introduce security holes, and I am assuming that the files are static (which may be wrong here). (*) or the Location .. + setHandler jakarta-servlet configuration variation - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow downloads through mod_jk on Windows XP
On 05/07/2012 06:06 PM, André Warnier wrote: Andy Wang wrote: Hi all, We've had a number of cases of people reporting to us that file downloads are slow when passed through tomcat and I've not been able to reproduce the problem on Linux but finally was provided a windows XP VM that was able to reproduce the problem. This is Apache 2.2.22 and mod_jk 1.2.32 with tomcat 5.0.30 (yeah, I know this is ancient. I'll try with something newer tomorrow). I have two URLs configured both with an identical iso file (roughly 450MB) in size. When I go through the URL configured to run directly through Apache the download speeds on the file are around 30MB/s which is reasonable for what I would expect on the network connection between the client and the web server. The second URL is JkMount'ed through to tomcat and download the same file drops down to 5-6 MB/s (starts at 10-ish MB/s). The one thing I haven't tried configuring is upping the ajp packet size because the version of tomcat I was provided doesn't support that, but otherwise the operating system is tuned so the TcpWindowSize is 131072. Unfortunately, this is windows, and both apache and tomcat are local on the server so I can't sniff the packets between the two systems with wireshark. I don't see anything on the mod_jk workers.properties configuration that deals with buffer sizes. Hoping someone has some ideas on what to look at that might help tweak this thing to perform similarly to Apache direct downloads. Considering your setup, it should not be too hard to set up a download of the same file file directly from Tomcat (through its HTTP Connector), to compare that with your two previous ways. This way, you could make sure if it is Tomcat, or the mod_jk/AJP link which is the issue. Also, still considering your setup, it should be possible to configure things so that these file downloads are handled directly by Apache httpd, since that seems to satisfy your expectations. mod_jk JkMount/JkUnMount rules (*) should make that possible, no ? Have to be a bit careful not to introduce security holes, and I am assuming that the files are static (which may be wrong here). (*) or the Location .. + setHandler jakarta-servlet configuration variation Thanks for the http connector idea. I forgot about that. The primary reason why i'm using tomcat to download a static file is really for testing purposes to confirm performance between mod_jk and direct apache. we have servlets that stream content files that see the same massive performance hit so in our actual use case it's not static files :(. I'm thinking this would be a valid test to help at least tweak mod_jk to it's potential. We've checked and double checked the buffering code of the servlets and it all looks fine AND the performance is fine on Linux and the speed characteristics are identical to serving static files through tomcat + mod_jk so I'm hoping that it's an apples to apples comparison. Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
