[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weijin updated TS-880: -- Attachment: performance.try3.diff Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.1 Attachments: performance.try3.diff, performance_2nd_req.diff, performance_2nd_req_try2.diff If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mohan_zl updated TS-880: Attachment: performance_2nd_req_another_version.patch Thank you William for your work, we tested your first diff file, and found ts will crash, so we write another patch. Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.1 Attachments: performance.try3.diff, performance_2nd_req.diff, performance_2nd_req_another_version.patch, performance_2nd_req_try2.diff If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Hedstrom updated TS-880: - Backport to Version: (was: 3.0.2) Fix Version/s: (was: 3.1.0) 3.1.1 I think we should spend some more time on this, it feels like the first approach (moving origin VConn's to the client VConn's ethread) makes the most sense (and clearly has the best performance characteristics. Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.1 Attachments: performance_2nd_req.diff, performance_2nd_req_try2.diff If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Bardwell updated TS-880: Attachment: performance_2nd_req_try2.diff New patch based on waking up event loop cross thread. If this is good, it might need to be ported to other event loop styles, as it is epoll only right now. Also on machines that support it an eventfd would supposedly be faster than a pipe. Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.0 Attachments: performance_2nd_req.diff, performance_2nd_req_try2.diff If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Bardwell updated TS-880: Attachment: performance_2nd_req.diff This fixes the problem for me. The name for my 'fix-it' function may not be very helpful. This may only be needed for epoll, I am not sure, and it may be bad for non-epoll, I am not sure. Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.0 Attachments: performance_2nd_req.diff If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-880) Major performance problem with second request on same keep-alive connection
[ https://issues.apache.org/jira/browse/TS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Hedstrom updated TS-880: - Fix Version/s: 3.1.0 Assignee: William Bardwell Major performance problem with second request on same keep-alive connection --- Key: TS-880 URL: https://issues.apache.org/jira/browse/TS-880 Project: Traffic Server Issue Type: Bug Components: HTTP Affects Versions: 3.0.0, 2.1.4 Environment: 32-bit TS on linux, using epoll. With proxy.config.http.keep_alive_enabled_out = 1 Reporter: William Bardwell Assignee: William Bardwell Fix For: 3.1.0 If I load the same URL through TS twice in a row to a server with keep-alives turned on I get really slow performance on the second request. E.g. (loading 212M over loopback with uncachable content, but results are similar with cachable content): % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 140M 0 0:00:01 0:00:01 --:--:-- 142M* % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 212M 100 212M0 0 6397k 0 0:00:34 0:00:34 --:--:-- 6400k* If I turn off proxy.config.http.keep_alive_enabled_out the problem goes away, it can also be partially solved by making proxy.config.io.max_buffer_size really big (but it needs to be bigger for larger content, and that supports the comment below, and proves that it isn't the origin server's fault.) When I turn on debug messages it looks like the IO loop for the second request only wakes up every 10ms (when an epoll_wait times out) and then it reads and writes, for the first request it goes much faster without those pauses. My theory is that the issue is related to the the fact that the outgoing connection fd gets added to the epoll fd for the thread handling the first request and the first user agent fd is added to the same epoll fd, and it stays on that epoll fd. But the new user agent request is on a different epoll fd which I assume means is tied to a different thread. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
