Hi all,
When I tested randomrw on my cluster through filebench (running ceph
0.94.5) , one of the osds was marked down. but I could still get the
process with ps command.
So I checked the log fiile and found follow message:
---------------------------------------------------->
2016-01-07 02:41:02.104124 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.5 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104156 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.6 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104168 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.7 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104182 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.8 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104194 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.12 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104208 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.15 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104226 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.16 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:02.104253 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.17 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:55.104035)
2016-01-07 02:41:03.104394 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.3 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104441 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.4 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104451 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.5 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104459 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.6 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104467 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.7 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104495 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.8 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104503 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.12 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104512 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.15 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104526 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.16 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
2016-01-07 02:41:03.104541 7fa9ae4cb700 -1 osd.11 1672 heartbeat_check:
no reply from osd.17 since back 2016-01-07 02:40:49.365340 front
2016-01-07 02:40:49.365340 (cutoff 2016-01-07 02:40:56.104394)
....
2016-01-07 02:56:17.340268 7fa98b99d700 0 -- 10.0.19.68:6816/10289
submit_message osd_op_reply(201270 1000005e069.0000046e [write
0~4194304] v1679'6462 uv6462 ondisk = 0) v6 remote, 10.0.3.68:0/49739,
failed lossy con, dropping message 0x30f84fc0
2016-01-07 02:56:17.886032 7fa9ae4cb700 0 log_channel(cluster) log
[WRN] : 1 slow requests, 1 included below; oldest blocked for > 9.802397
secs
2016-01-07 02:56:17.886195 7fa9ae4cb700 0 log_channel(cluster) log
[WRN] : slow request 9.802397 seconds old, received at 2016-01-07
02:56:08.083416: osd_op(client.501311.0:201273 1000005e069.00000471
[write 0~4194304] 7.ea64f958 RETRY=1 snapc 1=[]
ondisk+retry+write+known_if_redirected e1679) currently waiting for
subops from 3,6
2016-01-07 02:56:18.886521 7fa9ae4cb700 0 log_channel(cluster) log
[WRN] : 1 slow requests, 1 included below; oldest blocked for >
10.802942 secs
2016-01-07 02:56:18.886626 7fa9ae4cb700 0 log_channel(cluster) log
[WRN] : slow request 10.802942 seconds old, received at 2016-01-07
02:56:08.083416: osd_op(client.501311.0:201273 1000005e069.00000471
[write 0~4194304] 7.ea64f958 RETRY=1 snapc 1=[]
ondisk+retry+write+known_if_redirected e1679) currently waiting for
subops from 3,6
2016-01-07 03:37:27.066275 7f5f537007e0 0 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 105993
2016-01-07 03:37:27.066295 7f5f537007e0 0 turning on heap profiler with
prefix /var/log/ceph//osd.11.profile
2016-01-07 03:37:32.188191 7f5f537007e0 0
filestore(/var/lib/ceph/osd/ceph-11) lock_fsid failed to lock
/var/lib/ceph/osd/ceph-11/fsid, is another ceph-osd still running? (11)
Resource temporarily unavailable
2016-01-07 03:37:32.188225 7f5f537007e0 -1 osd.11 0 OSD::pre_init:
object store '/var/lib/ceph/osd/ceph-11' is currently in use. (Is
ceph-osd already running?)
2016-01-07 03:37:32.188233 7f5f537007e0 -1 ** ERROR: osd pre_init
failed: (16) Device or resource busy
<----------------------------------------------------
As listed above, this osd may have restarted failed automatically. I
manually restarted this process using 'service ceph restart osd.xx', the
process was still alive.
Then ,I debug this process with gdb:
---------------------------------------------------->
(gdb) info threads
142 Thread 0x7fa9b4ba6700 (LWP 10293) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
141 Thread 0x7fa9b43a5700 (LWP 10320) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
140 Thread 0x7fa9b32fc700 (LWP 10321) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
139 Thread 0x7fa9b1cd2700 (LWP 10329) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
138 Thread 0x7fa9b14d1700 (LWP 10330) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
137 Thread 0x7fa9b0cd0700 (LWP 10331) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
136 Thread 0x7fa9b04cf700 (LWP 10332) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
135 Thread 0x7fa9afcce700 (LWP 10333) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
134 Thread 0x7fa9af4cd700 (LWP 10334) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
133 Thread 0x7fa9aeccc700 (LWP 10335) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
132 Thread 0x7fa9ae4cb700 (LWP 10336) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
131 Thread 0x7fa9adcca700 (LWP 10337) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
130 Thread 0x7fa9acd74700 (LWP 10917) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
129 Thread 0x7fa9ac573700 (LWP 10918) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
128 Thread 0x7fa9abd72700 (LWP 10921) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
127 Thread 0x7fa9ab571700 (LWP 10922) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
126 Thread 0x7fa9aad70700 (LWP 10923) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
125 Thread 0x7fa9aa56f700 (LWP 10924) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
124 Thread 0x7fa9a9d6e700 (LWP 10925) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
123 Thread 0x7fa9a956d700 (LWP 10926) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
122 Thread 0x7fa9a8d6c700 (LWP 10927) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
121 Thread 0x7fa9a856b700 (LWP 10928) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
120 Thread 0x7fa9989b7700 (LWP 11808) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
119 Thread 0x7fa9981b6700 (LWP 11809) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
118 Thread 0x7fa9979b5700 (LWP 11810) 0x00007fa9b5ce8113 in poll ()
from /lib64/libc.so.6
117 Thread 0x7fa9971b4700 (LWP 11811) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
116 Thread 0x7fa9969b3700 (LWP 11812) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
115 Thread 0x7fa9959b1700 (LWP 11814) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
114 Thread 0x7fa9951b0700 (LWP 11815) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
113 Thread 0x7fa9949af700 (LWP 11816) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
112 Thread 0x7fa9941ae700 (LWP 11817) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
111 Thread 0x7fa9931ac700 (LWP 11819) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
110 Thread 0x7fa9929ab700 (LWP 11820) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
109 Thread 0x7fa9919a9700 (LWP 11822) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
108 Thread 0x7fa9911a8700 (LWP 11823) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
107 Thread 0x7fa9909a7700 (LWP 11824) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
106 Thread 0x7fa9901a6700 (LWP 11825) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
105 Thread 0x7fa98f9a5700 (LWP 11826) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
104 Thread 0x7fa98f1a4700 (LWP 11827) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
103 Thread 0x7fa98e9a3700 (LWP 11828) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
102 Thread 0x7fa98e1a2700 (LWP 11829) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
101 Thread 0x7fa98d9a1700 (LWP 11830) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
100 Thread 0x7fa98d1a0700 (LWP 11831) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
99 Thread 0x7fa98c99f700 (LWP 11832) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
98 Thread 0x7fa98c19e700 (LWP 11833) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
97 Thread 0x7fa98b99d700 (LWP 11834) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
96 Thread 0x7fa98b19c700 (LWP 11835) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
95 Thread 0x7fa98a99b700 (LWP 11836) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
94 Thread 0x7fa98a19a700 (LWP 11837) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
93 Thread 0x7fa989999700 (LWP 11838) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
92 Thread 0x7fa989198700 (LWP 11839) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
91 Thread 0x7fa988997700 (LWP 11840) 0x00007fa9b6f6da0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
90 Thread 0x7fa988196700 (LWP 11841) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
89 Thread 0x7fa987995700 (LWP 11842) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
88 Thread 0x7fa987194700 (LWP 11843) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
87 Thread 0x7fa986892700 (LWP 11845) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
86 Thread 0x7fa986091700 (LWP 11846) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
85 Thread 0x7fa985890700 (LWP 11847) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
84 Thread 0x7fa984313700 (LWP 11893) 0x00007fa9b5ce8113 in poll ()
from /lib64/libc.so.6
83 Thread 0x7fa97d66b700 (LWP 14038) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
82 Thread 0x7fa9961b2700 (LWP 105678) 0x00007fa9b5ce8113 in poll ()
from /lib64/libc.so.6
81 Thread 0x7fa9921aa700 (LWP 105679) 0x00007fa9b5ce8113 in poll ()
from /lib64/libc.so.6
80 Thread 0x7fa9939ad700 (LWP 105680) 0x00007fa9b5ce8113 in poll ()
from /lib64/libc.so.6
79 Thread 0x7fa9828d7700 (LWP 105697) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
78 Thread 0x7fa981e56700 (LWP 105745) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
77 Thread 0x7fa97f1e1700 (LWP 105749) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
76 Thread 0x7fa982058700 (LWP 105758) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
75 Thread 0x7fa9a41de700 (LWP 105768) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
74 Thread 0x7fa9a57fb700 (LWP 105769) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
73 Thread 0x7fa98380f700 (LWP 105772) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
72 Thread 0x7fa9830e8700 (LWP 105775) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
71 Thread 0x7fa9a2730700 (LWP 105776) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
70 Thread 0x7fa9a56fa700 (LWP 105778) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
69 Thread 0x7fa980c44700 (LWP 106167) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
68 Thread 0x7fa97e707700 (LWP 106168) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
67 Thread 0x7fa984c6b700 (LWP 106169) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
66 Thread 0x7fa984726700 (LWP 106170) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
65 Thread 0x7fa984625700 (LWP 106171) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
64 Thread 0x7fa984524700 (LWP 106172) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
63 Thread 0x7fa983b12700 (LWP 106173) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
62 Thread 0x7fa983a11700 (LWP 106174) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
61 Thread 0x7fa98330a700 (LWP 106178) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
60 Thread 0x7fa982fe7700 (LWP 106179) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
59 Thread 0x7fa9827d6700 (LWP 106180) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
58 Thread 0x7fa98235b700 (LWP 106181) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
57 Thread 0x7fa982159700 (LWP 106182) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
56 Thread 0x7fa981c54700 (LWP 106183) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
55 Thread 0x7fa981b53700 (LWP 106184) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
54 Thread 0x7fa981a52700 (LWP 106185) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
53 Thread 0x7fa981951700 (LWP 106186) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
52 Thread 0x7fa981850700 (LWP 106187) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
51 Thread 0x7fa98174f700 (LWP 106188) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
50 Thread 0x7fa98164e700 (LWP 106189) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
49 Thread 0x7fa98154d700 (LWP 106190) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
48 Thread 0x7fa98144c700 (LWP 106191) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
47 Thread 0x7fa98134b700 (LWP 106192) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
46 Thread 0x7fa98124a700 (LWP 106193) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
45 Thread 0x7fa980d94700 (LWP 106207) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
44 Thread 0x7fa98073f700 (LWP 106210) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
43 Thread 0x7fa98053d700 (LWP 106211) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
42 Thread 0x7fa980038700 (LWP 106214) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
41 Thread 0x7fa97ff37700 (LWP 106215) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
40 Thread 0x7fa97f7f1700 (LWP 106217) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
39 Thread 0x7fa97f4ee700 (LWP 106219) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
38 Thread 0x7fa97f2ec700 (LWP 106221) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
37 Thread 0x7fa97e3e4700 (LWP 106223) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
36 Thread 0x7fa97e2c3700 (LWP 106224) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
35 Thread 0x7fa984f8e700 (LWP 106232) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
34 Thread 0x7fa980b43700 (LWP 106241) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
33 Thread 0x7fa9a317e700 (LWP 106259) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
32 Thread 0x7fa9829d8700 (LWP 106260) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
31 Thread 0x7fa9a4927700 (LWP 113181) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
30 Thread 0x7fa9a3fdc700 (LWP 113186) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
29 Thread 0x7fa984d6c700 (LWP 113193) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
28 Thread 0x7fa9a2f7c700 (LWP 113203) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
27 Thread 0x7fa98350c700 (LWP 79532) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
26 Thread 0x7fa97f6f0700 (LWP 87703) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
25 Thread 0x7fa981f57700 (LWP 87704) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
24 Thread 0x7fa97db2c700 (LWP 88370) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
23 Thread 0x7fa982cdb700 (LWP 88371) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
22 Thread 0x7fa98225a700 (LWP 88372) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
21 Thread 0x7fa980139700 (LWP 88373) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
20 Thread 0x7fa9a45f1700 (LWP 88378) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
19 Thread 0x7fa983910700 (LWP 88379) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
18 Thread 0x7fa97fe36700 (LWP 88387) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
17 Thread 0x7fa98370e700 (LWP 88397) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
16 Thread 0x7fa9a2e7b700 (LWP 88789) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
15 Thread 0x7fa982bda700 (LWP 88790) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
14 Thread 0x7fa98063e700 (LWP 88791) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
13 Thread 0x7fa97e4f5700 (LWP 88792) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
12 Thread 0x7fa9a504b700 (LWP 88793) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
11 Thread 0x7fa98508f700 (LWP 88795) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
10 Thread 0x7fa97c8a3700 (LWP 88796) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
9 Thread 0x7fa9a3dce700 (LWP 88797) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
8 Thread 0x7fa97dc2d700 (LWP 88798) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
7 Thread 0x7fa97caa5700 (LWP 88800) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
6 Thread 0x7fa984e8d700 (LWP 88841) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
5 Thread 0x7fa98043c700 (LWP 90830) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
4 Thread 0x7fa9a622f700 (LWP 90833) 0x00007fa9b6f702e4 in
__lll_lock_wait () from /lib64/libpthread.so.0
3 Thread 0x7fa9a5b51700 (LWP 90845) 0x00007fa9b77326ab in
base::internal::SpinLockDelay(int volatile*, int, int) () from
/usr/lib64/libtcmalloc.so.4
2 Thread 0x7fa97c9a4700 (LWP 90851) 0x00007fa9b6f6d63c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 Thread 0x7fa9b820e7e0 (LWP 10292) 0x00007fa9b6f6a2ad in
pthread_join () from /lib64/libpthread.so.0
(gdb) thread 13
[Switching to thread 13 (Thread 0x7fa97e4f5700 (LWP 88792))]#0
0x00007fa9b77326ab in base::internal::SpinLockDelay(int volatile*, int,
int) () from /usr/lib64/libtcmalloc.so.4
(gdb) bt
#0 0x00007fa9b77326ab in base::internal::SpinLockDelay(int volatile*,
int, int) () from /usr/lib64/libtcmalloc.so.4
#1 0x00007fa9b772f9bc in SpinLock::SlowLock() () from
/usr/lib64/libtcmalloc.so.4
#2 0x00007fa9b772c0f1 in ?? () from /usr/lib64/libtcmalloc.so.4
#3 0x00007fa9b7725626 in MallocHook::InvokeNewHookSlow(void const*,
unsigned long) () from /usr/lib64/libtcmalloc.so.4
#4 0x00007fa9b7732d73 in tc_new () from /usr/lib64/libtcmalloc.so.4
#5 0x0000000000b2b982 in ceph::log::Log::create_entry (this=<value
optimized out>, level=0, subsys=27) at log/Log.cc:175
#6 0x0000000000c03075 in Pipe::fault (this=0x3b53d800, onread=<value
optimized out>) at msg/simple/Pipe.cc:1392
#7 0x0000000000c12114 in Pipe::reader (this=0x3b53d800) at
msg/simple/Pipe.cc:1674
#8 0x0000000000c1606d in Pipe::Reader::entry (this=<value optimized
out>) at msg/simple/Pipe.h:50
#9 0x00007fa9b6f69a51 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa9b5cf193d in clone () from /lib64/libc.so.6
(gdb)
<-------------
This is not the first time for me to find above problem.
Thanks,
wangsongbo
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
