Hi,
yesterday we had had the problem that one of our cluster clients
remounted a rbd device in read-only mode. We found this[1] stack trace
in the logs. We investigated further and found similar traces on all
other machines that are using the rbd kernel module. It seems to me that
whenever there is a swapping situation on a client those I/O errors occur.
Is there anything we can do or is this something that needs to be fixed
in the code?
client setup:
# uname -v
#1 SMP Debian 3.16.7-ckt2-1~bpo70+1 (2014-12-08)
# rbd -v
ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
# lsb_release -r
Release: 6.0.10
Cheers,
Sebastian
[1]
Feb 17 22:52:25 six kernel: [2401866.069932] kworker/6:1: page
allocation failure: order:1, mode:0x204020
Feb 17 22:52:25 six kernel: [2401866.069939] CPU: 6 PID: 4474 Comm:
kworker/6:1 Tainted: G W 3.16.0-0.bpo.4-amd64 #1 Debian
3.16.7-ckt2-1~bpo70+1
Feb 17 22:52:25 six kernel: [2401866.069940] Hardware name: Dell Inc.
PowerEdge R710/0PV9DG, BIOS 1.3.6 10/30/2009
Feb 17 22:52:25 six kernel: [2401866.069947] Workqueue: rbd1
rbd_request_workfn [rbd]
Feb 17 22:52:25 six kernel: [2401866.069949] 0000000000000000
0000000000000001 ffffffff81541f8f 0000000000204020
Feb 17 22:52:25 six kernel: [2401866.069951] ffffffff811519ed
0000000000000001 ffff88032fff2c00 0000000000000002
Feb 17 22:52:25 six kernel: [2401866.069953] ffffffff818e24d0
0000000100000002 ffff88032fff2c08 0000000000000092
Feb 17 22:52:25 six kernel: [2401866.069955] Call Trace:
Feb 17 22:52:25 six kernel: [2401866.069963] [<ffffffff81541f8f>] ?
dump_stack+0x41/0x51
Feb 17 22:52:25 six kernel: [2401866.069969] [<ffffffff811519ed>] ?
warn_alloc_failed+0xfd/0x160
Feb 17 22:52:25 six kernel: [2401866.069972] [<ffffffff8115606f>] ?
__alloc_pages_nodemask+0x91f/0xbb0
Feb 17 22:52:25 six kernel: [2401866.069977] [<ffffffff8119fe00>] ?
kmem_getpages+0x60/0x110
Feb 17 22:52:25 six kernel: [2401866.069979] [<ffffffff811a1648>] ?
fallback_alloc+0x158/0x220
Feb 17 22:52:25 six kernel: [2401866.069981] [<ffffffff811a1f44>] ?
kmem_cache_alloc+0x1a4/0x1e0
Feb 17 22:52:25 six kernel: [2401866.069987] [<ffffffffa0627889>] ?
ceph_osdc_alloc_request+0x69/0x320 [libceph]
Feb 17 22:52:25 six kernel: [2401866.069989] [<ffffffffa065f53b>] ?
rbd_osd_req_create.isra.17+0x7b/0x190 [rbd]
Feb 17 22:52:25 six kernel: [2401866.069992] [<ffffffffa0661fd5>] ?
rbd_img_request_fill+0x2b5/0x900 [rbd]
Feb 17 22:52:25 six kernel: [2401866.069995] [<ffffffffa0663485>] ?
rbd_request_workfn+0x235/0x350 [rbd]
Feb 17 22:52:25 six kernel: [2401866.070000] [<ffffffff810878fc>] ?
process_one_work+0x15c/0x450
Feb 17 22:52:25 six kernel: [2401866.070003] [<ffffffff81088b52>] ?
worker_thread+0x112/0x540
Feb 17 22:52:25 six kernel: [2401866.070005] [<ffffffff81088a40>] ?
create_and_start_worker+0x60/0x60
Feb 17 22:52:25 six kernel: [2401866.070008] [<ffffffff8108f511>] ?
kthread+0xc1/0xe0
Feb 17 22:52:25 six kernel: [2401866.070010] [<ffffffff8108f450>] ?
flush_kthread_worker+0xb0/0xb0
Feb 17 22:52:25 six kernel: [2401866.070013] [<ffffffff815483bc>] ?
ret_from_fork+0x7c/0xb0
Feb 17 22:52:25 six kernel: [2401866.070015] [<ffffffff8108f450>] ?
flush_kthread_worker+0xb0/0xb0
Feb 17 22:52:25 six kernel: [2401866.070017] Mem-Info:
Feb 17 22:52:25 six kernel: [2401866.070018] Node 0 Normal per-cpu:
Feb 17 22:52:25 six kernel: [2401866.070020] CPU 0: hi: 186, btch:
31 usd: 46
Feb 17 22:52:25 six kernel: [2401866.070021] CPU 1: hi: 186, btch:
31 usd: 168
Feb 17 22:52:25 six kernel: [2401866.070023] CPU 2: hi: 186, btch:
31 usd: 154
Feb 17 22:52:25 six kernel: [2401866.070024] CPU 3: hi: 186, btch:
31 usd: 177
Feb 17 22:52:25 six kernel: [2401866.070025] CPU 4: hi: 186, btch:
31 usd: 128
Feb 17 22:52:25 six kernel: [2401866.070026] CPU 5: hi: 186, btch:
31 usd: 152
Feb 17 22:52:25 six kernel: [2401866.070028] CPU 6: hi: 186, btch:
31 usd: 76
Feb 17 22:52:25 six kernel: [2401866.070029] CPU 7: hi: 186, btch:
31 usd: 109
Feb 17 22:52:25 six kernel: [2401866.070030] CPU 8: hi: 186, btch:
31 usd: 183
Feb 17 22:52:25 six kernel: [2401866.070031] CPU 9: hi: 186, btch:
31 usd: 180
Feb 17 22:52:25 six kernel: [2401866.070033] CPU 10: hi: 186, btch:
31 usd: 149
Feb 17 22:52:25 six kernel: [2401866.070034] CPU 11: hi: 186, btch:
31 usd: 170
Feb 17 22:52:25 six kernel: [2401866.070035] CPU 12: hi: 186, btch:
31 usd: 182
Feb 17 22:52:25 six kernel: [2401866.070036] CPU 13: hi: 186, btch:
31 usd: 169
Feb 17 22:52:25 six kernel: [2401866.070038] CPU 14: hi: 186, btch:
31 usd: 157
Feb 17 22:52:25 six kernel: [2401866.070039] CPU 15: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.070040] Node 1 DMA per-cpu:
Feb 17 22:52:25 six kernel: [2401866.070041] CPU 0: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070043] CPU 1: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070044] CPU 2: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070045] CPU 3: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070046] CPU 4: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070047] CPU 5: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070048] CPU 6: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070049] CPU 7: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070050] CPU 8: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070052] CPU 9: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070053] CPU 10: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070054] CPU 11: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070055] CPU 12: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070056] CPU 13: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070057] CPU 14: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070058] CPU 15: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.070059] Node 1 DMA32 per-cpu:
Feb 17 22:52:25 six kernel: [2401866.070061] CPU 0: hi: 186, btch:
31 usd: 21
Feb 17 22:52:25 six kernel: [2401866.070062] CPU 1: hi: 186, btch:
31 usd: 38
Feb 17 22:52:25 six kernel: [2401866.070063] CPU 2: hi: 186, btch:
31 usd: 173
Feb 17 22:52:25 six kernel: [2401866.070064] CPU 3: hi: 186, btch:
31 usd: 117
Feb 17 22:52:25 six kernel: [2401866.070065] CPU 4: hi: 186, btch:
31 usd: 87
Feb 17 22:52:25 six kernel: [2401866.070067] CPU 5: hi: 186, btch:
31 usd: 55
Feb 17 22:52:25 six kernel: [2401866.070068] CPU 6: hi: 186, btch:
31 usd: 138
Feb 17 22:52:25 six kernel: [2401866.070069] CPU 7: hi: 186, btch:
31 usd: 101
Feb 17 22:52:25 six kernel: [2401866.070071] CPU 8: hi: 186, btch:
31 usd: 89
Feb 17 22:52:25 six kernel: [2401866.070072] CPU 9: hi: 186, btch:
31 usd: 20
Feb 17 22:52:25 six kernel: [2401866.070073] CPU 10: hi: 186, btch:
31 usd: 183
Feb 17 22:52:25 six kernel: [2401866.070075] CPU 11: hi: 186, btch:
31 usd: 103
Feb 17 22:52:25 six kernel: [2401866.070076] CPU 12: hi: 186, btch:
31 usd: 159
Feb 17 22:52:25 six kernel: [2401866.070077] CPU 13: hi: 186, btch:
31 usd: 164
Feb 17 22:52:25 six kernel: [2401866.070078] CPU 14: hi: 186, btch:
31 usd: 168
Feb 17 22:52:25 six kernel: [2401866.070080] CPU 15: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.070080] Node 1 Normal per-cpu:
Feb 17 22:52:25 six kernel: [2401866.070082] CPU 0: hi: 186, btch:
31 usd: 172
Feb 17 22:52:25 six kernel: [2401866.070083] CPU 1: hi: 186, btch:
31 usd: 162
Feb 17 22:52:25 six kernel: [2401866.070084] CPU 2: hi: 186, btch:
31 usd: 197
Feb 17 22:52:25 six kernel: [2401866.070085] CPU 3: hi: 186, btch:
31 usd: 165
Feb 17 22:52:25 six kernel: [2401866.070087] CPU 4: hi: 186, btch:
31 usd: 172
Feb 17 22:52:25 six kernel: [2401866.070088] CPU 5: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.070089] CPU 6: hi: 186, btch:
31 usd: 173
Feb 17 22:52:25 six kernel: [2401866.070090] CPU 7: hi: 186, btch:
31 usd: 185
Feb 17 22:52:25 six kernel: [2401866.070092] CPU 8: hi: 186, btch:
31 usd: 171
Feb 17 22:52:25 six kernel: [2401866.070093] CPU 9: hi: 186, btch:
31 usd: 184
Feb 17 22:52:25 six kernel: [2401866.070094] CPU 10: hi: 186, btch:
31 usd: 156
Feb 17 22:52:25 six kernel: [2401866.070095] CPU 11: hi: 186, btch:
31 usd: 154
Feb 17 22:52:25 six kernel: [2401866.070097] CPU 12: hi: 186, btch:
31 usd: 202
Feb 17 22:52:25 six kernel: [2401866.070098] CPU 13: hi: 186, btch:
31 usd: 126
Feb 17 22:52:25 six kernel: [2401866.070099] CPU 14: hi: 186, btch:
31 usd: 174
Feb 17 22:52:25 six kernel: [2401866.070100] CPU 15: hi: 186, btch:
31 usd: 165
Feb 17 22:52:25 six kernel: [2401866.070104] active_anon:754337
inactive_anon:196881 isolated_anon:0
Feb 17 22:52:25 six kernel: [2401866.070104] active_file:1153206
inactive_file:537573 isolated_file:0
Feb 17 22:52:25 six kernel: [2401866.070104] unevictable:1908 dirty:453
writeback:0 unstable:0
Feb 17 22:52:25 six kernel: [2401866.070104] free:125203
slab_reclaimable:219609 slab_unreclaimable:35283
Feb 17 22:52:25 six kernel: [2401866.070104] mapped:20010 shmem:1497
pagetables:26565 bounce:0
Feb 17 22:52:25 six kernel: [2401866.070104] free_cma:0
Feb 17 22:52:25 six kernel: [2401866.070107] Node 0 Normal free:56624kB
min:45380kB low:56724kB high:68068kB active_anon:1445584kB
inactive_anon:305496kB active_file:2272332kB inactive_file:1423180kB
unevictable:5196kB isolated(anon):0kB isolated(file):0kB
present:6291456kB managed:6203124kB mlocked:5196kB dirty:1112kB
writeback:0kB mapped:51488kB shmem:3016kB slab_reclaimable:484828kB
slab_unreclaimable:91440kB kernel_stack:3008kB pagetables:71084kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:222
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.070112] lowmem_reserve[]: 0 0 0 0
Feb 17 22:52:25 six kernel: [2401866.070114] Node 1 DMA free:15904kB
min:116kB low:144kB high:172kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15904kB mlocked:0kB dirty:0kB
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? yes
Feb 17 22:52:25 six kernel: [2401866.070118] lowmem_reserve[]: 0 3227
5955 5955
Feb 17 22:52:25 six kernel: [2401866.070120] Node 1 DMA32 free:67160kB
min:24176kB low:30220kB high:36264kB active_anon:925144kB
inactive_anon:234688kB active_file:1353980kB inactive_file:444908kB
unevictable:1748kB isolated(anon):0kB isolated(file):0kB
present:3381860kB managed:3307208kB mlocked:1748kB dirty:468kB
writeback:0kB mapped:17888kB shmem:1436kB slab_reclaimable:201584kB
slab_unreclaimable:23428kB kernel_stack:9728kB pagetables:18816kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:107
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.070125] lowmem_reserve[]: 0 0 2727 2727
Feb 17 22:52:25 six kernel: [2401866.070127] Node 1 Normal free:361124kB
min:20436kB low:25544kB high:30652kB active_anon:646620kB
inactive_anon:247340kB active_file:986512kB inactive_file:282204kB
unevictable:688kB isolated(anon):0kB isolated(file):0kB
present:2883584kB managed:2793468kB mlocked:664kB dirty:232kB
writeback:0kB mapped:10664kB shmem:1536kB slab_reclaimable:192024kB
slab_unreclaimable:26264kB kernel_stack:1120kB pagetables:16360kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.070132] lowmem_reserve[]: 0 0 0 0
Feb 17 22:52:25 six kernel: [2401866.070134] Node 0 Normal: 13053*4kB
(UM) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 1*4096kB (R) = 56308kB
Feb 17 22:52:25 six kernel: [2401866.070141] Node 1 DMA: 0*4kB 0*8kB
0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB
(U) 1*2048kB (R) 3*4096kB (M) = 15904kB
Feb 17 22:52:25 six kernel: [2401866.070149] Node 1 DMA32: 15791*4kB
(UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 1*4096kB (R) = 67260kB
Feb 17 22:52:25 six kernel: [2401866.070155] Node 1 Normal: 89383*4kB
(U) 0*8kB 0*16kB 1*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB
1*1024kB (R) 1*2048kB (R) 0*4096kB = 361084kB
Feb 17 22:52:25 six kernel: [2401866.070164] Node 0 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 17 22:52:25 six kernel: [2401866.070165] Node 1 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 17 22:52:25 six kernel: [2401866.070166] 1693043 total pagecache pages
Feb 17 22:52:25 six kernel: [2401866.070168] 0 pages in swap cache
Feb 17 22:52:25 six kernel: [2401866.070169] Swap cache stats: add 0,
delete 0, find 0/0
Feb 17 22:52:25 six kernel: [2401866.070170] Free swap = 23390604kB
Feb 17 22:52:25 six kernel: [2401866.070171] Total swap = 23390604kB
Feb 17 22:52:25 six kernel: [2401866.070172] 3143224 pages RAM
Feb 17 22:52:25 six kernel: [2401866.070173] 0 pages HighMem/MovableOnly
Feb 17 22:52:25 six kernel: [2401866.070173] 22529 pages reserved
Feb 17 22:52:25 six kernel: [2401866.070174] 0 pages hwpoisoned
Feb 17 22:52:25 six kernel: [2401866.070178] rbd: rbd1: write 1000 at
5bab0000 result -12
Feb 17 22:52:25 six kernel: [2401866.070179] end_request: I/O error, dev
rbd1, sector 3003776
Feb 17 22:52:25 six kernel: [2401866.070299] Aborting journal on device
rbd1-8.
Feb 17 22:52:25 six kernel: [2401866.077852] EXT4-fs error (device
rbd1): ext4_journal_check_start:56: Detected aborted journal
Feb 17 22:52:25 six kernel: [2401866.078031] EXT4-fs (rbd1): Remounting
filesystem read-only
Feb 17 22:52:25 six kernel: [2401866.078139] kworker/6:1: page
allocation failure: order:1, mode:0x204020
Feb 17 22:52:25 six kernel: [2401866.078143] CPU: 6 PID: 4474 Comm:
kworker/6:1 Tainted: G W 3.16.0-0.bpo.4-amd64 #1 Debian
3.16.7-ckt2-1~bpo70+1
Feb 17 22:52:25 six kernel: [2401866.078145] Hardware name: Dell Inc.
PowerEdge R710/0PV9DG, BIOS 1.3.6 10/30/2009
Feb 17 22:52:25 six kernel: [2401866.078150] Workqueue: rbd1
rbd_request_workfn [rbd]
Feb 17 22:52:25 six kernel: [2401866.078152] 0000000000000000
0000000000000001 ffffffff81541f8f 0000000000204020
Feb 17 22:52:25 six kernel: [2401866.078154] ffffffff811519ed
0000000000000001 ffff88032fff2c00 0000000000000002
Feb 17 22:52:25 six kernel: [2401866.078156] ffffffff818e24d0
0000000100000002 ffff88032fff2c08 0000000000000092
Feb 17 22:52:25 six kernel: [2401866.078159] Call Trace:
Feb 17 22:52:25 six kernel: [2401866.078167] [<ffffffff81541f8f>] ?
dump_stack+0x41/0x51
Feb 17 22:52:25 six kernel: [2401866.078173] [<ffffffff811519ed>] ?
warn_alloc_failed+0xfd/0x160
Feb 17 22:52:25 six kernel: [2401866.078176] [<ffffffff8115606f>] ?
__alloc_pages_nodemask+0x91f/0xbb0
Feb 17 22:52:25 six kernel: [2401866.078180] [<ffffffff8119fe00>] ?
kmem_getpages+0x60/0x110
Feb 17 22:52:25 six kernel: [2401866.078183] [<ffffffff811a1648>] ?
fallback_alloc+0x158/0x220
Feb 17 22:52:25 six kernel: [2401866.078185] [<ffffffff811a1f44>] ?
kmem_cache_alloc+0x1a4/0x1e0
Feb 17 22:52:25 six kernel: [2401866.078190] [<ffffffffa0627889>] ?
ceph_osdc_alloc_request+0x69/0x320 [libceph]
Feb 17 22:52:25 six kernel: [2401866.078193] [<ffffffffa065f53b>] ?
rbd_osd_req_create.isra.17+0x7b/0x190 [rbd]
Feb 17 22:52:25 six kernel: [2401866.078196] [<ffffffffa0661fd5>] ?
rbd_img_request_fill+0x2b5/0x900 [rbd]
Feb 17 22:52:25 six kernel: [2401866.078198] [<ffffffffa0663485>] ?
rbd_request_workfn+0x235/0x350 [rbd]
Feb 17 22:52:25 six kernel: [2401866.078204] [<ffffffff810878fc>] ?
process_one_work+0x15c/0x450
Feb 17 22:52:25 six kernel: [2401866.078206] [<ffffffff81088b52>] ?
worker_thread+0x112/0x540
Feb 17 22:52:25 six kernel: [2401866.078208] [<ffffffff81088a40>] ?
create_and_start_worker+0x60/0x60
Feb 17 22:52:25 six kernel: [2401866.078211] [<ffffffff8108f511>] ?
kthread+0xc1/0xe0
Feb 17 22:52:25 six kernel: [2401866.078213] [<ffffffff8108f450>] ?
flush_kthread_worker+0xb0/0xb0
Feb 17 22:52:25 six kernel: [2401866.078217] [<ffffffff815483bc>] ?
ret_from_fork+0x7c/0xb0
Feb 17 22:52:25 six kernel: [2401866.078219] [<ffffffff8108f450>] ?
flush_kthread_worker+0xb0/0xb0
Feb 17 22:52:25 six kernel: [2401866.078220] Mem-Info:
Feb 17 22:52:25 six kernel: [2401866.078221] Node 0 Normal per-cpu:
Feb 17 22:52:25 six kernel: [2401866.078223] CPU 0: hi: 186, btch:
31 usd: 46
Feb 17 22:52:25 six kernel: [2401866.078225] CPU 1: hi: 186, btch:
31 usd: 168
Feb 17 22:52:25 six kernel: [2401866.078226] CPU 2: hi: 186, btch:
31 usd: 177
Feb 17 22:52:25 six kernel: [2401866.078227] CPU 3: hi: 186, btch:
31 usd: 177
Feb 17 22:52:25 six kernel: [2401866.078228] CPU 4: hi: 186, btch:
31 usd: 164
Feb 17 22:52:25 six kernel: [2401866.078230] CPU 5: hi: 186, btch:
31 usd: 152
Feb 17 22:52:25 six kernel: [2401866.078231] CPU 6: hi: 186, btch:
31 usd: 76
Feb 17 22:52:25 six kernel: [2401866.078232] CPU 7: hi: 186, btch:
31 usd: 109
Feb 17 22:52:25 six kernel: [2401866.078233] CPU 8: hi: 186, btch:
31 usd: 182
Feb 17 22:52:25 six kernel: [2401866.078235] CPU 9: hi: 186, btch:
31 usd: 180
Feb 17 22:52:25 six kernel: [2401866.078236] CPU 10: hi: 186, btch:
31 usd: 149
Feb 17 22:52:25 six kernel: [2401866.078237] CPU 11: hi: 186, btch:
31 usd: 170
Feb 17 22:52:25 six kernel: [2401866.078238] CPU 12: hi: 186, btch:
31 usd: 170
Feb 17 22:52:25 six kernel: [2401866.078240] CPU 13: hi: 186, btch:
31 usd: 169
Feb 17 22:52:25 six kernel: [2401866.078241] CPU 14: hi: 186, btch:
31 usd: 157
Feb 17 22:52:25 six kernel: [2401866.078242] CPU 15: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.078243] Node 1 DMA per-cpu:
Feb 17 22:52:25 six kernel: [2401866.078244] CPU 0: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078246] CPU 1: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078247] CPU 2: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078248] CPU 3: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078249] CPU 4: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078250] CPU 5: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078251] CPU 6: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078252] CPU 7: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078253] CPU 8: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078255] CPU 9: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078256] CPU 10: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078257] CPU 11: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078258] CPU 12: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078259] CPU 13: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078260] CPU 14: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078261] CPU 15: hi: 0, btch:
1 usd: 0
Feb 17 22:52:25 six kernel: [2401866.078262] Node 1 DMA32 per-cpu:
Feb 17 22:52:25 six kernel: [2401866.078264] CPU 0: hi: 186, btch:
31 usd: 21
Feb 17 22:52:25 six kernel: [2401866.078265] CPU 1: hi: 186, btch:
31 usd: 38
Feb 17 22:52:25 six kernel: [2401866.078266] CPU 2: hi: 186, btch:
31 usd: 173
Feb 17 22:52:25 six kernel: [2401866.078267] CPU 3: hi: 186, btch:
31 usd: 150
Feb 17 22:52:25 six kernel: [2401866.078269] CPU 4: hi: 186, btch:
31 usd: 87
Feb 17 22:52:25 six kernel: [2401866.078270] CPU 5: hi: 186, btch:
31 usd: 55
Feb 17 22:52:25 six kernel: [2401866.078271] CPU 6: hi: 186, btch:
31 usd: 138
Feb 17 22:52:25 six kernel: [2401866.078272] CPU 7: hi: 186, btch:
31 usd: 101
Feb 17 22:52:25 six kernel: [2401866.078273] CPU 8: hi: 186, btch:
31 usd: 89
Feb 17 22:52:25 six kernel: [2401866.078274] CPU 9: hi: 186, btch:
31 usd: 20
Feb 17 22:52:25 six kernel: [2401866.078276] CPU 10: hi: 186, btch:
31 usd: 183
Feb 17 22:52:25 six kernel: [2401866.078277] CPU 11: hi: 186, btch:
31 usd: 103
Feb 17 22:52:25 six kernel: [2401866.078278] CPU 12: hi: 186, btch:
31 usd: 159
Feb 17 22:52:25 six kernel: [2401866.078279] CPU 13: hi: 186, btch:
31 usd: 164
Feb 17 22:52:25 six kernel: [2401866.078280] CPU 14: hi: 186, btch:
31 usd: 168
Feb 17 22:52:25 six kernel: [2401866.078281] CPU 15: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.078282] Node 1 Normal per-cpu:
Feb 17 22:52:25 six kernel: [2401866.078284] CPU 0: hi: 186, btch:
31 usd: 172
Feb 17 22:52:25 six kernel: [2401866.078285] CPU 1: hi: 186, btch:
31 usd: 162
Feb 17 22:52:25 six kernel: [2401866.078286] CPU 2: hi: 186, btch:
31 usd: 199
Feb 17 22:52:25 six kernel: [2401866.078288] CPU 3: hi: 186, btch:
31 usd: 165
Feb 17 22:52:25 six kernel: [2401866.078289] CPU 4: hi: 186, btch:
31 usd: 172
Feb 17 22:52:25 six kernel: [2401866.078290] CPU 5: hi: 186, btch:
31 usd: 176
Feb 17 22:52:25 six kernel: [2401866.078291] CPU 6: hi: 186, btch:
31 usd: 173
Feb 17 22:52:25 six kernel: [2401866.078292] CPU 7: hi: 186, btch:
31 usd: 185
Feb 17 22:52:25 six kernel: [2401866.078294] CPU 8: hi: 186, btch:
31 usd: 171
Feb 17 22:52:25 six kernel: [2401866.078295] CPU 9: hi: 186, btch:
31 usd: 184
Feb 17 22:52:25 six kernel: [2401866.078296] CPU 10: hi: 186, btch:
31 usd: 156
Feb 17 22:52:25 six kernel: [2401866.078297] CPU 11: hi: 186, btch:
31 usd: 154
Feb 17 22:52:25 six kernel: [2401866.078299] CPU 12: hi: 186, btch:
31 usd: 182
Feb 17 22:52:25 six kernel: [2401866.078300] CPU 13: hi: 186, btch:
31 usd: 126
Feb 17 22:52:25 six kernel: [2401866.078301] CPU 14: hi: 186, btch:
31 usd: 174
Feb 17 22:52:25 six kernel: [2401866.078302] CPU 15: hi: 186, btch:
31 usd: 165
Feb 17 22:52:25 six kernel: [2401866.078306] active_anon:755093
inactive_anon:196881 isolated_anon:0
Feb 17 22:52:25 six kernel: [2401866.078306] active_file:1144456
inactive_file:540838 isolated_file:0
Feb 17 22:52:25 six kernel: [2401866.078306] unevictable:1908 dirty:453
writeback:0 unstable:0
Feb 17 22:52:25 six kernel: [2401866.078306] free:130001
slab_reclaimable:219609 slab_unreclaimable:35283
Feb 17 22:52:25 six kernel: [2401866.078306] mapped:20010 shmem:1497
pagetables:26565 bounce:0
Feb 17 22:52:25 six kernel: [2401866.078306] free_cma:0
Feb 17 22:52:25 six kernel: [2401866.078309] Node 0 Normal free:75816kB
min:45380kB low:56724kB high:68068kB active_anon:1448608kB
inactive_anon:305496kB active_file:2237784kB inactive_file:1435800kB
unevictable:5196kB isolated(anon):0kB isolated(file):0kB
present:6291456kB managed:6203124kB mlocked:5196kB dirty:1112kB
writeback:0kB mapped:51488kB shmem:3016kB slab_reclaimable:484828kB
slab_unreclaimable:91440kB kernel_stack:3008kB pagetables:71084kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:71
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.078315] lowmem_reserve[]: 0 0 0 0
Feb 17 22:52:25 six kernel: [2401866.078317] Node 1 DMA free:15904kB
min:116kB low:144kB high:172kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15996kB managed:15904kB mlocked:0kB dirty:0kB
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? yes
Feb 17 22:52:25 six kernel: [2401866.078321] lowmem_reserve[]: 0 3227
5955 5955
Feb 17 22:52:25 six kernel: [2401866.078323] Node 1 DMA32 free:67160kB
min:24176kB low:30220kB high:36264kB active_anon:925144kB
inactive_anon:234688kB active_file:1353528kB inactive_file:445348kB
unevictable:1748kB isolated(anon):0kB isolated(file):0kB
present:3381860kB managed:3307208kB mlocked:1748kB dirty:468kB
writeback:0kB mapped:17888kB shmem:1436kB slab_reclaimable:201584kB
slab_unreclaimable:23428kB kernel_stack:9728kB pagetables:18816kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:214
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.078328] lowmem_reserve[]: 0 0 2727 2727
Feb 17 22:52:25 six kernel: [2401866.078330] Node 1 Normal free:361124kB
min:20436kB low:25544kB high:30652kB active_anon:646620kB
inactive_anon:247340kB active_file:986512kB inactive_file:282204kB
unevictable:688kB isolated(anon):0kB isolated(file):0kB
present:2883584kB managed:2793468kB mlocked:664kB dirty:232kB
writeback:0kB mapped:10664kB shmem:1536kB slab_reclaimable:192024kB
slab_unreclaimable:26264kB kernel_stack:1120kB pagetables:16360kB
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
Feb 17 22:52:25 six kernel: [2401866.078334] lowmem_reserve[]: 0 0 0 0
Feb 17 22:52:25 six kernel: [2401866.078336] Node 0 Normal: 17586*4kB
(UEM) 74*8kB (UEM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 1*4096kB (R) = 75032kB
Feb 17 22:52:25 six kernel: [2401866.078343] Node 1 DMA: 0*4kB 0*8kB
0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB
(U) 1*2048kB (R) 3*4096kB (M) = 15904kB
Feb 17 22:52:25 six kernel: [2401866.078351] Node 1 DMA32: 15791*4kB
(UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 1*4096kB (R) = 67260kB
Feb 17 22:52:25 six kernel: [2401866.078357] Node 1 Normal: 89352*4kB
(U) 0*8kB 0*16kB 1*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB
1*1024kB (R) 1*2048kB (R) 0*4096kB = 360960kB
Feb 17 22:52:25 six kernel: [2401866.078366] Node 0 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 17 22:52:25 six kernel: [2401866.078368] Node 1 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 17 22:52:25 six kernel: [2401866.078369] 1687591 total pagecache pages
Feb 17 22:52:25 six kernel: [2401866.078370] 0 pages in swap cache
Feb 17 22:52:25 six kernel: [2401866.078372] Swap cache stats: add 0,
delete 0, find 0/0
Feb 17 22:52:25 six kernel: [2401866.078372] Free swap = 23390604kB
Feb 17 22:52:25 six kernel: [2401866.078373] Total swap = 23390604kB
Feb 17 22:52:25 six kernel: [2401866.078374] 3143224 pages RAM
Feb 17 22:52:25 six kernel: [2401866.078375] 0 pages HighMem/MovableOnly
Feb 17 22:52:25 six kernel: [2401866.078376] 22529 pages reserved
Feb 17 22:52:25 six kernel: [2401866.078377] 0 pages hwpoisoned
Feb 17 22:52:25 six kernel: [2401866.078380] rbd: rbd1: write 1000 at 0
result -12
Feb 17 22:52:25 six kernel: [2401866.078382] end_request: I/O error, dev
rbd1, sector 0
Feb 17 22:52:25 six kernel: [2401866.078449] Buffer I/O error on device
rbd1, logical block 0
Feb 17 22:52:25 six kernel: [2401866.078515] lost page write due to I/O
error on rbd1
--
Mit freundlichen Grüßen
Sebastian Köhler
------------------------------------
Alfahosting GmbH
Ankerstraße 3b
06108 Halle
Germany
Steuernummer: 110/115/41765
HRB : 214733 AG Stendal
Ust-IdNr.: DE232969203
Geschäftsführer: Moritz Bartsch
Email: i...@alfahosting.de
Internet: www.alfahosting.de
TEL +49 (0345) 279 58 304
FAX +49 (0345) 680 04 99
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
