Re: Performance problem with NIO under high concurrency
As an experimentation, I started Tomcat with
“-Dorg.apache.tomcat.util.net.NioSelectorShared=false”, and
“selectorPool.maxSelectors” set to 1000, the same as the number of threads.
This problem didn’t happen with that setting. Even with
“selectorPool.maxSelectors” set to 1, it was noticeably slow, but wasn’t stuck.
I’m surprised that there’s only one shared selector by default. I’m reading the
code to get a better understanding of the SelectorPool implementation. I’d
appreciate any insights you may have around this.
Thanks,
Thuc
On 8/31/17, 9:09 AM, "Thuc Nguyen" <tnguy...@linkedin.com> wrote:
Hi Chris,
Thanks for the quick response.
Yes, the clients were being throttled. These throttled requests were slow
to start with and there was no noticeable difference in the download speed when
the problem occurred. The smaller download started out OK. But after 10-15
successful serial requests it hit the problem. Then it remained slow until the
big downloads completed or got canceled.
The connector configuration:
We’ll upgrade to 8.0.46 and see if the problem can be reproduced.
Thanks again,
Thuc
On 8/31/17, 8:51 AM, "Christopher Schultz" <ch...@christopherschultz.net>
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Thuc,
On 8/31/17 11:25 AM, Thuc Nguyen wrote:
> We run JFrog Artifactory which is fronted by Tomcat 8.0.32. We
> recently upgraded from Tomcat 7.0.56. Since the upgrade,
> Artifactory occasionally slows to a crawl.
Any chance of using the latest Tomcat 8.0.46?
> We could reproduce this problem by downloading a large (1GB) file
> concurrently from 500 different machines. However, to avoid
> exceeding the network bandwidth, we throttled the downloads at
> 100KB per second.
You throttled the clients?
> While these 500 requests were being processed, we sent a request
> to download a smaller file (1MB, no throttling). The smaller
> download successfully went through several times, then got “stuck”.
> The download speed was reduced to a few KB per second.
Over all connections, or just the single "small" requests? Was is
stuck permanently? Meaning, if it got "stuck" one time (timed out?),
would any later requests succeed?
> We couldn’t reproduce this problem on 7.0.56. We kept 8.0.32 but
> changed the connector to BIO and couldn’t reproduce it either. We
> suspect that it has something to do with switching from BIO in
> 7.0.56 to NIO in 8.0.32.
It's possible, but you are using a fairly old version of Tomcat (~18
months).
> Has anyone run into this problem before?
>
> Server.xml, a thread dump and Catalina MBean values are attached.
> Tomcat version:
Can you please post your configuration from server.xml?
- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJZqDCMAAoJEBzwKT+lPKRYOYMP/RJzkQueb1YORwm6hN0I8u6n
idyq+byMzM+JmEM+uVprrBqFKwCGZEYMqKcrf/cirLlpr1F1bpiDyjWXxX6p2zZt
IfHKH1oMYa4EpCXGn4PDLag9cCqA1sr44MOzkQ9PvQFJGFLo12exwQDd++EL1YxH
DQmMsGh+xGmnaaK9EAJ85e9Jl6t+OHGF0655V8gt5MxsWfDGM+U8i2RDJln0HX0J
GWJCcs6ce6qzi82GYXf597QrKhF7x7lYrZMH2JbghxUMqB6GUw0uAyhGdYO5OfYz
wVUkB9dzDMwMli8UmmSclN/rF2msKL+RD7G72oNNCyxOmZc5oS4BvLVwLa5a7CCP
UPXuCLAXRg+JQ9C4MXGOXvhEKpkOKjaDFBsSSnavycPMSCfK9gCRXIU37DfIZFhN
1k/Jjq13VNaq6KgX2UkEbqwwuEmyjhXLuuz6KC714Hm6raeQEiaE88W2defvc6PG
179/bwXiMxhW6BXa2RCrhMHrqfw1gxIvqdwqoM1xEKIJhAhVD2AjJLpKA2AGJ1Tw
15nictwVIpmjYN13r0/j6UrT3KT6qlbSQT9d9FaErAqBaFWtdknqSe4ZDWcd9i59
Cg8PktKQSpr2dDijxGusydBc9QYzIS28ofLjZwSF2UXTW/rNAVVmtU+8O3y8zCTB
lXk0eAvyyWpVxkis8+Ft
=2s/6
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance problem with NIO under high concurrency
Hi Chris, Thanks for the quick response. Yes, the clients were being throttled. These throttled requests were slow to start with and there was no noticeable difference in the download speed when the problem occurred. The smaller download started out OK. But after 10-15 successful serial requests it hit the problem. Then it remained slow until the big downloads completed or got canceled. The connector configuration: We’ll upgrade to 8.0.46 and see if the problem can be reproduced. Thanks again, Thuc On 8/31/17, 8:51 AM, "Christopher Schultz" <ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thuc, On 8/31/17 11:25 AM, Thuc Nguyen wrote: > We run JFrog Artifactory which is fronted by Tomcat 8.0.32. We > recently upgraded from Tomcat 7.0.56. Since the upgrade, > Artifactory occasionally slows to a crawl. Any chance of using the latest Tomcat 8.0.46? > We could reproduce this problem by downloading a large (1GB) file > concurrently from 500 different machines. However, to avoid > exceeding the network bandwidth, we throttled the downloads at > 100KB per second. You throttled the clients? > While these 500 requests were being processed, we sent a request > to download a smaller file (1MB, no throttling). The smaller > download successfully went through several times, then got “stuck”. > The download speed was reduced to a few KB per second. Over all connections, or just the single "small" requests? Was is stuck permanently? Meaning, if it got "stuck" one time (timed out?), would any later requests succeed? > We couldn’t reproduce this problem on 7.0.56. We kept 8.0.32 but > changed the connector to BIO and couldn’t reproduce it either. We > suspect that it has something to do with switching from BIO in > 7.0.56 to NIO in 8.0.32. It's possible, but you are using a fairly old version of Tomcat (~18 months). > Has anyone run into this problem before? > > Server.xml, a thread dump and Catalina MBean values are attached. > Tomcat version: Can you please post your configuration from server.xml? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJZqDCMAAoJEBzwKT+lPKRYOYMP/RJzkQueb1YORwm6hN0I8u6n idyq+byMzM+JmEM+uVprrBqFKwCGZEYMqKcrf/cirLlpr1F1bpiDyjWXxX6p2zZt IfHKH1oMYa4EpCXGn4PDLag9cCqA1sr44MOzkQ9PvQFJGFLo12exwQDd++EL1YxH DQmMsGh+xGmnaaK9EAJ85e9Jl6t+OHGF0655V8gt5MxsWfDGM+U8i2RDJln0HX0J GWJCcs6ce6qzi82GYXf597QrKhF7x7lYrZMH2JbghxUMqB6GUw0uAyhGdYO5OfYz wVUkB9dzDMwMli8UmmSclN/rF2msKL+RD7G72oNNCyxOmZc5oS4BvLVwLa5a7CCP UPXuCLAXRg+JQ9C4MXGOXvhEKpkOKjaDFBsSSnavycPMSCfK9gCRXIU37DfIZFhN 1k/Jjq13VNaq6KgX2UkEbqwwuEmyjhXLuuz6KC714Hm6raeQEiaE88W2defvc6PG 179/bwXiMxhW6BXa2RCrhMHrqfw1gxIvqdwqoM1xEKIJhAhVD2AjJLpKA2AGJ1Tw 15nictwVIpmjYN13r0/j6UrT3KT6qlbSQT9d9FaErAqBaFWtdknqSe4ZDWcd9i59 Cg8PktKQSpr2dDijxGusydBc9QYzIS28ofLjZwSF2UXTW/rNAVVmtU+8O3y8zCTB lXk0eAvyyWpVxkis8+Ft =2s/6 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org server.xml Description: server.xml - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Performance problem with NIO under high concurrency
Hi, We run JFrog Artifactory which is fronted by Tomcat 8.0.32. We recently upgraded from Tomcat 7.0.56. Since the upgrade, Artifactory occasionally slows to a crawl. We could reproduce this problem by downloading a large (1GB) file concurrently from 500 different machines. However, to avoid exceeding the network bandwidth, we throttled the downloads at 100KB per second. While these 500 requests were being processed, we sent a request to download a smaller file (1MB, no throttling). The smaller download successfully went through several times, then got “stuck”. The download speed was reduced to a few KB per second. We couldn’t reproduce this problem on 7.0.56. We kept 8.0.32 but changed the connector to BIO and couldn’t reproduce it either. We suspect that it has something to do with switching from BIO in 7.0.56 to NIO in 8.0.32. Has anyone run into this problem before? Server.xml, a thread dump and Catalina MBean values are attached. Tomcat version: Server version: Apache Tomcat/8.0.32 Server built: Feb 2 2016 19:34:53 UTC Server number: 8.0.32.0 OS Name:Linux OS Version: 2.6.32-504.el6.x86_64 Architecture: amd64 JVM Version:1.8.0_72-b15 JVM Vendor: Oracle Corporation Thank you, Thuc Nguyen tomcat.tar.gz Description: tomcat.tar.gz - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
