bump
I have observed this crash on ubuntu with kernel 3.13 and centos with 3.16 as
well now.
rbd hangs, and iostat shows something similar to the Output below.
Micha Krause
Am 19.09.2014 um 09:22 schrieb Micha Krause:
Hi,
> I have build an NFS Server based on Sebastiens Blog Post here:
http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/
Im using Kernel 3.14-0.bpo.1-amd64 on Debian wheezy, the host is a VM on Vmware.
Using rsync im writing data via nfs from one client to this Server.
The NFS Server crashes multiple times per day, I can't even login to the Server
then.
After a reset, there is no kernel log about the crash, so I guess something is
blocking
all I/Os.
Ok, it seems that I just can't get a shell, but I can run commands via ssh
directly.
I was able to get the following informations:
dmesg:
[18102.981064] INFO: task nfsd:2769 blocked for more than 120 seconds.
[18102.981112] Not tainted 3.14-0.bpo.1-amd64 #1
[18102.981150] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[18102.981216] nfsd D ffff88003fc14340 0 2769 2 0x00000000
[18102.981218] ffff88003bac6e20 0000000000000046 0000000000000000
ffff88003d47ada0
[18102.981219] 0000000000014340 ffff88003ce31fd8 0000000000014340
ffff88003bac6e20
[18102.981221] ffff88003ce31728 ffff8800029539f0 7fffffffffffffff
7fffffffffffffff
[18102.981223] Call Trace:
[18102.981225] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
[18102.981231] [<ffffffffa04b0f92>] ? _xfs_buf_find+0xd2/0x280 [xfs]
[18102.981234] [<ffffffff8117fc2c>] ? kmem_cache_alloc+0x1bc/0x1f0
[18102.981236] [<ffffffff814f193c>] ? __down_common+0x97/0xea
[18102.981241] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs]
[18102.981243] [<ffffffff810aa697>] ? down+0x37/0x40
[18102.981247] [<ffffffffa04b0e02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
[18102.981252] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs]
[18102.981257] [<ffffffffa04b1215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
[18102.981263] [<ffffffffa04b2153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
[18102.981269] [<ffffffffa05161da>] ? xfs_trans_read_buf_map+0x34a/0x4f0 [xfs]
[18102.981275] [<ffffffffa05036f9>] ? xfs_imap_to_bp+0x69/0xf0 [xfs]
[18102.981281] [<ffffffffa0503bcd>] ? xfs_iread+0x7d/0x3f0 [xfs]
[18102.981284] [<ffffffff810e8939>] ? make_kgid+0x9/0x10
[18102.981286] [<ffffffff811b148e>] ? inode_init_always+0x10e/0x1d0
[18102.981292] [<ffffffffa04ba11a>] ? xfs_iget+0x2ba/0x810 [xfs]
[18102.981298] [<ffffffffa04fd9a6>] ? xfs_ialloc+0xe6/0x740 [xfs]
[18102.981305] [<ffffffffa04ca1ee>] ? kmem_zone_alloc+0x6e/0xf0 [xfs]
[18102.981311] [<ffffffffa04fe083>] ? xfs_dir_ialloc+0x83/0x300 [xfs]
[18102.981317] [<ffffffffa04c8e43>] ? xfs_trans_reserve+0x213/0x220 [xfs]
[18102.981323] [<ffffffffa04fe87e>] ? xfs_create+0x4fe/0x720 [xfs]
[18102.981329] [<ffffffffa04bfd02>] ? xfs_vn_mknod+0xd2/0x200 [xfs]
[18102.981331] [<ffffffff811a6b54>] ? vfs_create+0xe4/0x160
[18102.981335] [<ffffffffa0400d9e>] ? do_nfsd_create+0x53e/0x610 [nfsd]
[18102.981339] [<ffffffffa0407f4d>] ? nfsd3_proc_create+0x16d/0x250 [nfsd]
[18102.981342] [<ffffffffa03f9d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
[18102.981347] [<ffffffffa035dd64>] ? svc_process_common+0x354/0x690 [sunrpc]
[18102.981349] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
[18102.981353] [<ffffffffa035e3fb>] ? svc_process+0x10b/0x160 [sunrpc]
[18102.981359] [<ffffffffa03f96d7>] ? nfsd+0xb7/0x130 [nfsd]
[18102.981363] [<ffffffffa03f9620>] ? nfsd_destroy+0x70/0x70 [nfsd]
[18102.981365] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
[18102.981367] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
[18102.981369] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
[18102.981371] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
iostat:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 1.00 99.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
rbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
46.00 0.00 0.00 0.00 0.00 100.00
rbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12.00 0.00 0.00 0.00 0.00 100.00
rbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
136.00 0.00 0.00 0.00 0.00 100.00
rbd3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
rbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11.00 0.00 0.00 0.00 0.00 100.00
rbd5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
57.00 0.00 0.00 0.00 0.00 100.00
emcpowerig 0.00 0.00 0.00 0.00 0.00 0.00 0.00
32.00 0.00 0.00 0.00 0.00 100.00
emcpowerhq 0.00 0.00 0.00 0.00 0.00 0.00 0.00
38.00 0.00 0.00 0.00 0.00 100.00
(for some reason rbd6 and rbd7 are shown as emcpower in iostat, no idea why)
cat /sys/kernel/debug/ceph/*/*
have osdmap 39226
want next osdmap
epoch 10
mon0 10.210.32.11:6789
mon1 10.210.33.11:6789
mon2 10.210.34.11:6789
582277 osd19 37.e52208ae rb.0.1676178.2ae8944a.000000005ffd
write
582278 osd2 37.f52b9433 rb.0.1676178.2ae8944a.000000006000
write
582279 osd2 37.b3f0aae3 rb.0.1676178.2ae8944a.00000000641b
write
582280 osd28 37.d8768bba rb.0.1676178.2ae8944a.00000000641c
write
582282 osd29 37.a923b4c6 rb.0.1676178.2ae8944a.000000008032
write
582283 osd28 37.a2510620 rb.0.1676178.2ae8944a.000000008034
write
582289 osd18 37.c96bc19d rb.0.1345def.2ae8944a.0000001401cf
write
582290 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3
write
582291 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee
write
582292 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee
write
582293 osd2 37.d34be89b rb.0.1345def.2ae8944a.00000003c961
read
582294 osd4 37.c611292f rb.0.1345def.2ae8944a.00000007a19a
read
582295 osd20 37.b1ae9634 rb.0.1345def.2ae8944a.00000003def7
read
582296 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106
write
582297 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614
write
582298 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614
write
582299 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149
write
582300 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582301 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582302 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582303 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582304 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582305 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582306 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582307 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb
write
582308 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582309 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614
write
582310 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614
write
582311 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582312 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582313 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582314 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582315 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582316 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582317 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582318 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9
write
582319 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba
write
582320 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582321 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582322 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582323 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582324 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582325 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582326 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582327 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc
write
582328 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd
write
582329 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106
write
582330 osd23 37.834e04c8 rb.0.1689cf4.238e1f29.00000002485f
write
582331 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea
write
582332 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea
write
582333 osd29 37.e3741a18 rb.0.1689cf4.238e1f29.00000002c0f6
write
582334 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7
write
582335 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7
write
582336 osd26 37.3f51e7ac rb.0.1689cf4.238e1f29.0000000320e7
write
582337 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6
write
582338 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6
write
582339 osd18 37.e3d0fce6 rb.0.1689cf4.238e1f29.00000002005f
write
582340 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba
write
582341 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba
write
582342 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba
write
582343 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba
write
582344 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd
write
582345 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd
write
582346 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd
write
582347 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582348 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582349 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582350 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582351 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582352 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392
write
582353 osd4 37.1067846f rb.0.1676178.2ae8944a.00000003a43c
write
582354 osd26 37.ecccc929 rb.0.1676178.2ae8944a.00000003c703
write
582355 osd18 37.ed91673e rb.0.1676178.2ae8944a.00000003c704
write
582356 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1
write
582357 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1
write
582358 osd23 37.a050d40f rb.0.1676178.2ae8944a.00000003e0a4
write
582359 osd1 37.c3f1f60c rb.0.1676178.2ae8944a.00000003e24f
write
582360 osd18 37.8ad6d326 rb.0.1676178.2ae8944a.00000003e50b
write
582361 osd28 37.cdf2c9ba rb.0.1676178.2ae8944a.00000003e50c
write
582362 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d
write
582363 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d
write
582364 osd16 37.1eaec3ff rb.0.1676178.2ae8944a.00000000641a
write
582365 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc
write
582366 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc
write
582367 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc
write
582368 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c
write
582369 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c
write
582370 osd18 37.876fd4d7 rb.0.1676178.2ae8944a.000000008430
write
582371 osd19 37.e08b5547 rb.0.1676178.2ae8944a.000000008431
write
582372 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e
write
582373 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e
write
582374 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5
write
582375 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5
write
582376 osd17 37.82f3d395 rb.0.1676178.2ae8944a.000000016173
write
582377 osd19 37.9af0f044 rb.0.1676178.2ae8944a.000000016389
write
582378 osd29 37.6d121a86 rb.0.1676178.2ae8944a.00000002004a
write
582379 osd2 37.ee45629b rb.0.1689d69.238e1f29.00000000484e
write
582380 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd
write
582381 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd
write
582382 osd27 37.56c53271 rb.0.1689d69.238e1f29.00000000618d
write
582383 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc
write
582384 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc
write
582385 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc
write
582386 osd1 37.947c5bcc rb.0.1689d69.238e1f29.0000000080bd
write
582387 osd26 37.7ae1d612 rb.0.1689d69.238e1f29.000000008112
write
582388 osd18 37.9bf4cf3c rb.0.1689d69.238e1f29.0000000081eb
write
582389 osd1 37.9d5cf7f0 rb.0.1689d69.238e1f29.0000000081ec
write
582390 osd19 37.4f750fee rb.0.1689d69.238e1f29.0000000081f0
write
582391 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb
write
582392 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb
write
582393 osd26 37.94b385c rb.0.1689d69.238e1f29.00000000a1d1
write
582394 osd28 37.fd40607a rb.0.1689d69.238e1f29.000000022197
write
582395 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614
write
582396 osd27 37.c42eab1 rb.0.1689d69.238e1f29.000000036160
write
582397 osd2 37.2ac07662 rb.0.1689d69.238e1f29.00000001fffc
write
582398 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149
write
582399 osd18 37.5c0f9c25 rb.0.167614e.2ae8944a.00000002a133
write
582400 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8
write
582401 osd26 37.9be1215c rb.0.167614e.2ae8944a.00000000042b
write
582402 osd23 37.a64e7d0f rb.0.167614e.2ae8944a.00000000238f
write
582403 osd20 37.da28f8ab rb.0.167614e.2ae8944a.000000003ffe
write
582404 osd24 37.b7af1f40 rb.0.167614e.2ae8944a.000000004005
write
582405 osd20 37.cecda9b4 rb.0.167614e.2ae8944a.000000004b8a
write
582406 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd
write
582407 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd
write
582408 osd27 37.f29b1b01 rb.0.167614e.2ae8944a.000000006242
write
582409 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc
write
582410 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc
write
582411 osd24 37.87cc6a8e rb.0.167614e.2ae8944a.0000000082f2
write
582412 osd15 37.a91eb70d rb.0.167614e.2ae8944a.000000009ffb
write
582413 osd15 37.3b20fd05 rb.0.167614e.2ae8944a.00000000a03a
write
582414 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8
write
582415 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8
write
582416 osd26 37.f0bb2a2a rb.0.167614e.2ae8944a.0000000301cc
write
582417 osd26 37.3b4ebe12 rb.0.167614e.2ae8944a.0000000301cd
write
582418 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7
write
582419 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7
write
582420 osd27 37.3f46e175 rb.0.167614e.2ae8944a.00000003215c
write
582421 osd18 37.796dd926 rb.0.167614e.2ae8944a.00000003217a
write
582422 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6
write
582423 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6
write
582424 osd26 37.84b7afe9 rb.0.167614e.2ae8944a.000000034112
write
582425 osd24 37.f9a234b8 rb.0.167614e.2ae8944a.000000034113
write
582426 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5
write
582427 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5
write
582428 osd23 37.1ae6123b rb.0.167614e.2ae8944a.00000003611a
write
582429 osd18 37.8202a597 rb.0.167614e.2ae8944a.00000003611b
write
582430 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4
write
582431 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4
write
582432 osd4 37.7caa0b rb.0.167614e.2ae8944a.00000003800a
write
582433 osd4 37.5917feaf rb.0.167614e.2ae8944a.0000000380cd
write
582434 osd29 37.abe267c6 rb.0.167614e.2ae8944a.0000000380cf
write
582435 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3
write
582436 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3
write
582437 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f
write
582438 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f
write
582439 osd27 37.33f8b641 rb.0.167614e.2ae8944a.00000003cc91
write
582440 osd22 37.be13ad3d rb.0.167614e.2ae8944a.00000003e40d
write
582441 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035
write
582442 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035
write
582443 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3
write
582444 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3
write
582445 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3
write
582446 osd16 37.4800911f rb.0.165171e.238e1f29.00000003a085
write
582447 osd2 37.bdec20a3 rb.0.165171e.238e1f29.00000003a12c
write
582448 osd23 37.33d51508 rb.0.165171e.238e1f29.00000003a12d
write
582449 osd16 37.8a8ec527 rb.0.165171e.238e1f29.00000003a138
write
582450 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139
write
582451 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139
write
582452 osd15 37.7e4b4f9e rb.0.165171e.238e1f29.00000003a13a
write
582453 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b
write
582454 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b
write
582455 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c
write
582456 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c
write
582457 osd19 37.dcc1f244 rb.0.165171e.238e1f29.00000003a13d
write
582458 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e
write
582459 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e
write
582460 osd18 37.d7371b26 rb.0.165171e.238e1f29.00000003a13f
write
582461 osd18 37.89ec9be5 rb.0.165171e.238e1f29.00000003a140
write
582462 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2
write
582463 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2
write
582464 osd26 37.54a4fd50 rb.0.165171e.238e1f29.00000003bfe9
write
582465 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136
write
582466 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136
write
582467 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1
write
582468 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1
write
582469 osd24 37.685a8638 rb.0.165171e.238e1f29.00000003e08c
write
582470 osd26 37.adfd8b12 rb.0.165171e.238e1f29.00000003e14a
write
582471 osd2 37.a67386a2 rb.0.165171e.238e1f29.00000003e14b
write
582472 osd18 37.688ac754 rb.0.165171e.238e1f29.00000002802c
write
582473 osd15 37.d2bed74d rb.0.1676160.238e1f29.00000002000d
write
582474 osd23 37.9c9a1a8f rb.0.165171e.238e1f29.000000020002
write
epoch 39226
flags
pg_pool 0 pg_num 256 / 255
pg_pool 1 pg_num 128 / 127
pg_pool 4 pg_num 32 / 31
pg_pool 19 pg_num 512 / 511
pg_pool 25 pg_num 8 / 7
pg_pool 27 pg_num 1 / 0
pg_pool 28 pg_num 1 / 0
pg_pool 29 pg_num 1 / 0
pg_pool 30 pg_num 1 / 0
pg_pool 31 pg_num 1 / 0
pg_pool 32 pg_num 1 / 0
pg_pool 33 pg_num 1 / 0
pg_pool 34 pg_num 1 / 0
pg_pool 35 pg_num 2 / 1
pg_pool 36 pg_num 1 / 0
pg_pool 37 pg_num 64 / 63
pg_pool 40 pg_num 2 / 1
pg_pool 41 pg_num 1 / 0
osd0 10.210.33.22:6815 100% (exists, up)
osd1 10.210.33.22:6800 100% (exists, up)
osd2 10.210.33.22:6805 100% (exists, up)
osd3 10.210.32.22:6800 0% (doesn't exist)
osd4 10.210.33.22:6810 100% (exists, up)
osd5 10.210.33.22:6820 100% (doesn't exist)
osd6 10.210.33.22:6805 100% (doesn't exist)
osd7 10.210.33.22:6825 100% (doesn't exist)
osd8 10.210.33.22:6830 100% (doesn't exist)
osd9 10.210.33.22:6835 100% (doesn't exist)
osd10 10.210.33.22:6805 100% (doesn't exist)
osd11 10.210.33.22:6845 100% (doesn't exist)
osd12 10.210.33.22:6850 100% (doesn't exist)
osd13 10.210.33.22:6855 100% (doesn't exist)
osd14 10.210.33.22:6860 100% (doesn't exist)
osd15 10.210.32.23:6800 100% (exists, up)
osd16 10.210.32.23:6807 100% (exists, up)
osd17 10.210.32.23:6801 100% (exists, up)
osd18 10.210.32.23:6816 100% (exists, up)
osd19 10.210.32.23:6812 100% (exists, up)
osd20 10.210.34.21:6800 100% (exists, up)
osd21 10.210.34.21:6804 100% (exists, up)
osd22 10.210.34.21:6809 100% (exists, up)
osd23 10.210.34.21:6814 100% (exists, up)
osd24 10.210.34.21:6819 100% (exists, up)
osd25 10.210.33.21:6800 100% (exists, up)
osd26 10.210.33.21:6805 100% (exists, up)
osd27 10.210.33.21:6810 100% (exists, up)
osd28 10.210.33.21:6815 100% (exists, up)
osd29 10.210.33.21:6820 100% (exists, up)
osd30 10.210.33.22:6865 100% (doesn't exist)
osd31 10.210.33.22:6870 0% (doesn't exist)
osd32 10.210.33.21:6800 0% (doesn't exist)
I don't know how to interpret this, the doesn't exist lines are correct, these
osds where removed.
Why are they still known to the rbd client? The OSDs where removed before the
client was booted.
Micha Krause
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
