[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286338#comment-14286338 ] Leif Hedstrom commented on TS-1336: --- I spoke with God (jplevyak) on this issue recently. One suspicion here is that we're doing things either at too short timeout intervals, or scheduling things too frequently. Definitely worth looking into, and also, maybe TS-3313 will help with some of these issues. High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Labels: A Fix For: sometime On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 000 S0 0.0 0:00.05 kthreadd stracing the ET_TASK threads showed a repeating set of calls to
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14197006#comment-14197006
]
Leif Hedstrom commented on TS-1336:
---
[~shinrich] Have you seen this? Not talking about the small, normal CPU usage
that happens, but this obscene 60-100% CPU usage on an idle instance?
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: sometime
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
2 root 20 0 000 S0 0.0 0:00.05 kthreadd
stracing the ET_TASK threads showed a repeating set of calls to futex:
futex(0x946ca4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 255405471,
{1341604150, 0}, ) = -1
[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119270#comment-14119270 ] Leif Hedstrom commented on TS-1336: --- Any more feedback on this bug? The amount of CPU initially reported is absurdly large, but a few percent is still as expected I think. And now you should have an option to reduce the epoll timeout (not great, but might be enough?). High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Labels: A Fix For: sometime On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 000 S0 0.0 0:00.05 kthreadd stracing the ET_TASK threads showed a repeating set of calls to futex: futex(0x946ca4,
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875932#comment-13875932
]
Igor Galić commented on TS-1336:
We have a workaround in TS-2474 - but that's still not entirely fixed.
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: sometime
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
2 root 20 0 000 S0 0.0 0:00.05 kthreadd
stracing the ET_TASK threads showed a repeating set of calls to futex:
futex(0x946ca4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 255405471,
{1341604150, 0}, ) = -1 ETIMEDOUT (Connection timed out)
futex(0x946ce0, FUTEX_WAKE_PRIVATE, 1) = 0
I installed
[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863089#comment-13863089 ] Leif Hedstrom commented on TS-1336: --- With TS-1365, do we want to keep this open still? Can we move it out to later ? It would be nice to reduce some CPU usage without mucking with the epoll timeouts, but I'm not sure if anyone is willing to take on that work? High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Assignee: Leif Hedstrom Labels: A Fix For: 4.2.0 On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 000 S0 0.0 0:00.05 kthreadd stracing the ET_TASK threads showed a repeating set of calls to futex:
[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863306#comment-13863306 ] James Peach commented on TS-1336: - My preferred solution to this is to do exponential back off when the poll finds no events pending, and reset to the default fast poll rate when events are processed. This would be auto-tuning, which is preferable. On the other hand, the event poll doesn't look easily changed to support this, so I expect it is a fair amount of effort. High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Assignee: Leif Hedstrom Labels: A Fix For: 4.2.0 On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 000 S0
[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738885#comment-13738885 ] Reindl Harald commented on TS-1336: --- this is really a problem i have running a test-vm on my home-server which has way too high CPU load even if all is idle caused by ATS as well as the production ATS-VM at night without anything to do eats the same idle-cou as 10 identical vservers without ATS for now i need to diable ATS on my testserver because it makes power-management of the host impossible :-( High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Labels: A Fix For: 3.5.1 On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 0
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665134#comment-13665134
]
Igor Galić commented on TS-1336:
Here's the same probe from our omnios system:
{noformat}
igalic@omnios-x64 % sudo /usr/sbin/dtrace -n 'syscall:::entry /execname ==
traffic_server/ { @num[probefunc] = count(); }'
dtrace: description 'syscall:::entry ' matched 232 probes
^C
fcntl14
close15
open 15
write15
rusagesys35
nanosleep65
memcntl 137
yield 256
portfs 2217
lwp_park 16893178
igalic@omnios-x64 %
{noformat}
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: 3.3.4
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665149#comment-13665149
]
Igor Galić commented on TS-1336:
{preformat}
igalic@omnios-x64 % sudo /usr/sbin/plockstat -e60 -p $(pgrep traffic_server) |
gc++filt
0
Mutex block
Count nsec Lock Caller
---
1445 5878518 0x46f3050libc.so.1`__cond_timedwait+0xc8
{preformat}
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: 3.3.4
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
2 root 20 0 000 S0 0.0 0:00.05
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665165#comment-13665165
]
Igor Galić commented on TS-1336:
{noformat}
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
15562 jenkins 3.5 3.0 0.0 0.0 0.0 48 0.0 45 608 138 .5M 0 traffic_serv/6
15562 jenkins 3.4 3.1 0.0 0.0 0.0 48 0.0 45 607 135 .5M 0 traffic_serv/9
15562 jenkins 3.4 3.1 0.0 0.0 0.0 48 0.0 45 608 134 .5M 0 traffic_serv/7
15562 jenkins 3.4 3.0 0.0 0.0 0.0 48 0.0 45 608 133 .5M 0 traffic_serv/10
15562 jenkins 3.4 3.0 0.0 0.0 0.0 48 0.0 45 607 134 .5M 0 traffic_serv/8
15562 jenkins 3.4 3.0 0.0 0.0 0.0 48 0.0 45 607 133 .5M 0 traffic_serv/12
15562 jenkins 3.4 3.0 0.0 0.0 0.0 48 0.0 45 607 132 .5M 0 traffic_serv/11
15562 jenkins 3.3 3.0 0.0 0.0 0.0 48 0.0 45 608 132 .5M 0 traffic_serv/13
15562 jenkins 0.0 0.1 0.0 0.0 0.0 0.0 97 3.3 441 0 431 0 traffic_serv/1
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 12 0 traffic_serv/5
15562 jenkins 0.0 0.0 0.0 0.0 0.0 98 0.0 2.2 213 0 213 0 traffic_serv/16
15562 jenkins 0.0 0.0 0.0 0.0 0.0 98 0.0 2.2 194 0 194 0 traffic_serv/17
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 4 0 4 0 traffic_serv/4
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 2 0 traffic_serv/3
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 traffic_serv/18
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 traffic_serv/15
15562 jenkins 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 0 0 0 0 traffic_serv/14
15562 jenkins 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 2 0 traffic_serv/2
{noformat}
unfortunately, Illumos doesn't have named threads, so we don't kno which
threads are locking which :C
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: 3.3.4
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665405#comment-13665405
]
Igor Galić commented on TS-1336:
please note that the load average on the system is slowly rising: Even though
ATS isn't doing anything on this machine, after ~2 hours the load average was
46.~
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Labels: A
Fix For: 3.3.4
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
2 root 20 0 000 S0 0.0 0:00.05 kthreadd
stracing the ET_TASK threads showed a repeating set of calls to futex:
futex(0x946ca4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 255405471,
{1341604150, 0},
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13613067#comment-13613067
]
Leif Hedstrom commented on TS-1336:
---
There was a suggestion that we have a too short rescheduling / timeout setting.
Who wants to look at this? :)
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Fix For: 3.3.3
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
2 root 20 0 000 S0 0.0 0:00.05 kthreadd
stracing the ET_TASK threads showed a repeating set of calls to futex:
futex(0x946ca4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 255405471,
{1341604150, 0}, ) = -1 ETIMEDOUT (Connection timed out)
futex(0x946ce0,
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13489022#comment-13489022
]
Igor Galić commented on TS-1336:
My observation with trunk on FreeBSD (9.1/amd64, 2 CPUs, ca 10s recording):
{noformat}
igalic@daemonix ~ % w
8:53PM up 51 mins, 0 users, load averages: 7.65, 7.28, 6.30
USER TTY FROM LOGIN@ IDLE WHAT
igalic@daemonix ~ % sudo dtrace -n 'syscall:::entry /execname ==
traffic_server/ { @num[probefunc] = count(); }'
Password:
dtrace: description 'syscall:::entry ' matched 1046 probes
dtrace: buffer size lowered to 2m
dtrace: aggregation size lowered to 2m
^C
read 1
write 1
open 2
getrusage 5
nanosleep10
msync79
kevent 1509
_umtx_op 12638
clock_gettime327793
igalic@daemonix ~ %
{noformat}
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
Fix For: 3.3.1
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics
[jira] [Commented] (TS-1336) High CPU Usage at idle
[ https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420549#comment-13420549 ] Martin Last commented on TS-1336: - Hi guys, I can confirm the issue on a 2 core HT sandy brdige box. I constantly have a load 2.3, while the proxy has little to none todo. Questions: - do you need more input to fix this issue - is there a patch that I can apply to get this sorted? - don't want to wait for 3.3.0/1 version. Many Thanks, Martin High CPU Usage at idle -- Key: TS-1336 URL: https://issues.apache.org/jira/browse/TS-1336 Project: Traffic Server Issue Type: Bug Components: Performance Affects Versions: 3.0.5, 3.0.2 Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16 cores with HT) Reporter: Greg Smolyn Fix For: 3.3.1 On this unloaded system, a very basic traffic server instance is using 180% CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60% each. top -H output: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1] 10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0] 10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING] 19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3 10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1 27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5 35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7 3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0 15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2 23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4 31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6 13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server 9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top 10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5] 10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9] 10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17] 10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18] 10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21] 1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init 2 root 20 0 000 S0 0.0 0:00.05 kthreadd stracing the ET_TASK threads showed
[jira] [Commented] (TS-1336) High CPU Usage at idle
[
https://issues.apache.org/jira/browse/TS-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13410070#comment-13410070
]
Peter commented on TS-1336:
---
I saw something identical; here's an strace of the LOGGING thread
futex(0x96fc64, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 9006185,
{1341898346, 0}, ) = -1 ETIMEDOUT (Connection timed out)
futex(0x96fc20, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x96fc64, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 9006187,
{1341898346, 0}, ) = -1 ETIMEDOUT (Connection timed out)
futex(0x96fc20, FUTEX_WAKE_PRIVATE, 1) = 0
I ultimately rebooted the box and the problem went away (Debian squeeze,
2.6.32-5; 100+ days of uptime).
High CPU Usage at idle
--
Key: TS-1336
URL: https://issues.apache.org/jira/browse/TS-1336
Project: Traffic Server
Issue Type: Bug
Components: Performance
Affects Versions: 3.0.5, 3.0.2
Environment: Ubuntu 12.04 server, amd64, Xenon E5520 (4-core, 16
cores with HT)
Reporter: Greg Smolyn
On this unloaded system, a very basic traffic server instance is using 180%
CPU, with 3 threads ET_TASK 0, ET_TASK 1, and LOGGING taking up about 60%
each.
top -H output:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
10723 traffics 20 0 1960m 113m 4168 R 61 0.4 9:11.27 [ET_TASK 1]
10722 traffics 20 0 1960m 113m 4168 R 60 0.4 8:41.61 [ET_TASK 0]
10720 traffics 20 0 1960m 113m 4168 S 59 0.4 8:49.19 [LOGGING]
19 root 20 0 000 R 15 0.0 898:45.74 ksoftirqd/3
10 root 20 0 000 S 15 0.0 930:16.92 ksoftirqd/1
27 root 20 0 000 S 14 0.0 893:18.41 ksoftirqd/5
35 root 20 0 000 S 14 0.0 888:54.41 ksoftirqd/7
3 root 20 0 000 S8 0.0 942:48.39 ksoftirqd/0
15 root 20 0 000 S7 0.0 906:40.98 ksoftirqd/2
23 root 20 0 000 S7 0.0 907:30.33 ksoftirqd/4
31 root 20 0 000 S7 0.0 898:13.05 ksoftirqd/6
13530 root 20 0 98.2m 3244 2572 S1 0.0 29:28.86 flip_server
9425 root 20 0 17568 1592 1060 R0 0.0 0:04.16 top
10689 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 5]
10693 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.51 [ET_NET 9]
10701 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.56 [ET_NET 17]
10702 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.53 [ET_NET 18]
10705 traffics 20 0 1960m 113m 4168 S0 0.4 0:00.54 [ET_NET 21]
1 root 20 0 24328 2256 1344 S0 0.0 0:02.53 init
