2011-07-03 13:38:57 -0600, cwillu:
> On Sun, Jul 3, 2011 at 1:09 PM, Stephane Chazelas
> <stephane_chaze...@yahoo.fr> wrote:
[...]
> > Now, on a few occasions (actually, most of the time), when I
> > rsynced the data (about 2.5TB) onto the external drive, the
> > system would crash after some time with "Out of memory and no
> > killable process". Basically, something in kernel was allocating
> > the whole memory, then oom mass killed everybody and crash.
[...]
> Look at the output of slabtop (should be installed by default, procfs
> package), before rsync for comparison, and during.
Hi,
so, no crash this time, but at the end of the rsync, there's a
whole chunk of memory that is no longer available to processes
(about 3.5G). As suggested by Carey, I watched /proc/slabinfo
during the rsync process (see below for a report of the most
significant ones over time).
Does that mean that if the system had had less than 3G of RAM, it
would have crashed?
I tried to reclaim the space without success. I had a process
allocate as much memory as it could. Then I unmounted the btrfs
fs that rsync was copying onto (the one on LUKS).
btrfs_inode_cache hardly changed. Then I tried to unmount the
source (the one on 3 hard disks and plenty of subvolumes).
umount hung. The FS disappeared from /proc/mounts. Here is the
backtrace:
[169270.268005] umount D ffff880145ebe770 0 24079 1290 0x00000004
[169270.268005] ffff880145ebe770 0000000000000086 0000000000000000
ffffffff8160b020
[169270.268005] 0000000000012840 ffff880123bc7fd8 ffff880123bc7fd8
0000000000012840
[169270.268005] ffff880145ebe770 ffff880123bc6010 00007fffac84f4a8
0000000100000000
[169270.268005] Call Trace:
[169270.268005] [<ffffffff81337d50>] ? rwsem_down_failed_common+0xda/0x10e
[169270.268005] [<ffffffff811aca63>] ? call_rwsem_down_write_failed+0x13/0x20
[169270.268005] [<ffffffff813376c7>] ? down_write+0x25/0x27
[169270.268005] [<ffffffff810ff6eb>] ? deactivate_super+0x30/0x3d
[169270.268005] [<ffffffff81114135>] ? sys_umount+0x2ea/0x315
[169270.268005] [<ffffffff8133d412>] ? system_call_fastpath+0x16/0x1b
iostat shows nothing being written to the drives.
extent_map delayed_node btrfs_inode_cache btrfs_free_space_cache
(in bytes)
09:40 131560 0 0 128
09:50 131560 0 2000 128 rsync started at 9:52
10:00 15832608 87963264 1465830000 325440
10:10 386056 85071168 1444656000 832960
10:20 237600 33988032 1549769000 1355584
10:30 23300288 145209600 487274000 2237056
10:40 24667104 148610016 506492000 2304448
10:50 22479776 139655808 511893000 2382464
11:00 4137672 1041696 4592000 2425344
11:10 5498592 3211776 12742000 2443648
11:20 4567904 1408896 9599000 2452608
11:30 2276736 1309824 4685000 2453696
11:40 2225696 421920 1987000 2455424
11:50 1971552 90432 383000 2466176
12:00 1761672 74016 327000 2469760
12:10 1939608 85824 401000 2473536
12:20 2136288 121824 551000 2479168
12:30 2367288 135648 619000 2486016
12:40 1380984 181152 911000 2485696
12:50 1053272 202176 1027000 2483712
13:00 1938200 219744 1152000 2491712
13:10 2037112 223776 1204000 2494528
13:20 1775664 249120 1244000 2497216
13:30 1704560 366048 1732000 2500608
13:40 1433344 468864 2460000 2501824
13:50 8553248 20505888 67332000 2503168
14:00 12682208 34351200 141968000 2494208
14:10 18488800 50282784 177803000 2500544
14:20 19435592 46767744 163582000 2505920
14:30 18734936 44863488 156501000 2507200
14:40 21865184 46053504 160185000 2484928
14:50 24457664 46473120 162446000 2499200
15:00 24401344 47700576 166723000 2502784
15:10 31390304 63426240 221179000 2521472
15:20 34594560 61365600 214243000 2524160
15:30 33836704 60934752 212695000 2526400
15:40 33358776 60598944 211455000 2528320
15:50 34909952 62583840 218492000 2526272
16:00 44326656 65875392 230123000 2529792
16:10 45840608 66373632 232114000 2532736
16:20 47848064 66577536 232048000 2535872
16:30 48013152 68888160 240651000 2536128
16:40 47594184 67766976 236241000 2536576
16:50 48144184 67739904 236120000 2542144
17:00 48005848 67639392 235298000 2544000
17:10 48253920 67661280 235376000 2537216
17:20 48857952 67612032 234995000 2536000
17:30 48514752 67611168 234986000 2535488
17:40 48436872 67609728 234924000 2534528
17:50 48902216 67765248 235654000 2542400
18:00 49055160 67763520 236020000 2542912
18:10 48749712 67727520 235740000 2550464
18:20 48631088 67705344 235557000 2553280
18:30 49101096 67777344 235713000 2555520
18:40 48609264 67782816 235601000 2558912
18:50 48480080 67808160 235595000 2561792
19:00 48661184 67830912 235680000 2563776
19:10 49004296 67826880 235609000 2565056
19:20 48530592 67819968 235510000 2565440
19:30 47806352 67827456 235904000 2569536
19:40 48563064 67806432 235902000 2572224
19:50 49107960 67845600 239902000 2577472
20:00 48902744 67828320 237062000 2577600
20:10 48921752 67897728 236337000 2578880
20:20 48713016 67927680 236237000 2579520
20:30 47901216 67961376 236228000 2580416
20:40 55052096 75482784 263292000 2586944
20:50 64494496 81030528 286325000 2594368
21:00 65381536 80284608 279964000 2598464
21:10 68131712 80154720 279335000 2598336
21:20 73455712 79982208 278654000 2586048
21:30 78164064 81536832 284003000 2604544
21:40 79013088 86614848 303401000 2615744
21:50 84289568 94296384 328520000 2626560
22:00 85835464 92268576 322902000 2627264
22:10 84338584 91186272 317134000 2627520
22:20 84768728 91129248 316551000 2625664
22:30 84280856 91147968 316611000 2634880
22:40 84117704 91176192 316714000 2637312
22:50 84217760 91232928 316791000 2640640
23:00 84345976 91288224 318467000 2642880
23:10 84431864 91296864 318490000 2643648
23:20 84579968 91302912 318509000 2644160
23:30 84381792 91284480 318288000 2644800
23:40 84323624 91298016 318010000 2645440
23:50 84220488 91296864 317799000 2648640
00:00 84217496 91293408 317572000 2651008
00:10 84178072 91276416 317235000 2653952
00:20 84208432 91290816 317230000 2656384
00:30 84245216 91312992 317310000 2657088
00:40 84235360 91314144 317064000 2657856
00:50 84151760 91332864 317116000 2658176
01:00 84146040 91344096 317184000 2658880
01:10 84119288 91358784 317238000 2659200
01:20 84158624 91373760 317277000 2663808
01:30 84247680 91378368 317292000 2665984
01:40 84209576 91393344 317344000 2669504
01:50 84286400 91410048 317402000 2660224
02:00 84312976 91406592 317390000 2656512
02:10 84228496 91423008 317442000 2665408
02:20 84065256 91506816 317703000 2668800
02:30 84093152 91589184 318030000 2671552
02:40 84038768 91686528 318343000 2672000
02:50 84134424 91713600 318460000 2680832
03:00 84183792 91728576 318508000 2684288
03:10 84259296 91728576 318505000 2688000
03:20 84237296 91743552 318557000 2690752
03:30 84212832 91759968 318609000 2691968
03:40 84150088 91778688 318661000 2692224
03:50 84133104 91738656 318552000 2692928
04:00 84129672 91728576 318533000 2693760
04:10 84178600 91711296 318446000 2701888
04:20 84164872 91726272 318497000 2704384
04:30 84231752 91717632 318467000 2708864
04:40 84260000 91711872 318447000 2714880
04:50 84261672 91707552 318432000 2714304
05:00 84204472 91709856 318435000 2714240
05:10 84128616 91732032 318487000 2715136
05:20 84111984 91736640 318555000 2715904
05:30 84145512 91742112 318552000 2710656
05:40 84111192 91740672 318561000 2719552
05:50 84169624 91755648 318600000 2722304
06:00 84047480 91791360 318704000 2724800
06:10 84107496 91874304 319036000 2727488
06:20 84129672 91968768 319336000 2727808
06:30 84144192 92054592 319610000 2728192
06:40 84245304 92050560 320572000 2728896
06:50 84257800 92036160 320522000 2729472
07:00 84299600 92050560 320572000 2733376
07:10 84148504 92037312 320521000 2736832
07:20 84120960 92013696 320427000 2739200
07:30 84097816 92000160 320142000 2742464
07:40 84112512 91984896 319954000 2743040
07:50 84197960 91976832 319751000 2744000
08:00 84180448 91979424 319679000 2744640
08:10 84199720 91983456 319626000 2746496
08:20 84152200 91989792 319615000 2745664
08:30 84309808 92004768 319653000 2755456
08:40 84288424 92022048 319683000 2758208
08:50 84162672 92040192 319675000 2761664
09:00 84128616 92099232 319874000 2765312
09:10 84160208 92192544 320177000 2767424
09:20 84299952 92249280 320383000 2765760
09:30 84265808 92241504 320329000 2765888
09:40 84335416 92254176 320314000 2766208
09:50 84231488 92255328 320331000 2771648
10:00 84155016 92266848 320366000 2773568
10:10 84077664 92279232 320418000 2776320
10:20 84151584 92290752 320484000 2774080
10:30 84245128 92305728 320509000 2768768
10:40 84226032 92284992 320442000 2761408
10:50 84265192 92292192 320462000 2765056
11:00 84308136 92285856 320440000 2770176
11:10 84273816 92281248 320408000 2771008
11:20 84210016 92292480 320460000 2775872
11:30 84137152 92278656 320408000 2780096
11:40 84097288 92301696 320476000 2785536
11:50 83999344 92389824 320791000 2791296
12:00 84162936 92478240 321127000 2791360
12:10 84116208 92541600 321512000 2768960
12:20 84727896 92524608 321464000 2763008
12:30 84868520 92582496 321797000 2772096
12:40 85114216 92604960 322010000 2774848
12:50 85450024 92558304 321832000 2778112
13:00 86018064 92632320 322221000 2780032
13:10 85731536 92576448 321966000 2781248
13:20 86218352 92672928 322370000 2781120
13:30 86143728 92668032 322394000 2781568
13:40 86400688 92681280 322335000 2781760
13:50 85993512 92759616 322693000 2785536
14:00 86842976 92859264 323995000 2788288
14:10 86913024 92849184 323884000 2790784
14:20 89733600 93937248 327207000 2794240
14:30 93420800 95194080 331519000 2795008
14:40 90696496 94583808 329243000 2795136
14:50 89758240 94404096 328002000 2795328
15:00 89516944 94354272 327777000 2796160
15:10 89768096 94390848 327930000 2798656
15:20 92897376 109204704 382426000 2803008
15:30 99914144 130731552 456126000 2770880
15:40 100707464 124378848 431912000 2775168
15:50 100372888 124369920 431846000 2777984
16:00 100679128 124373664 431858000 2779776
16:10 107601120 136025856 474697000 2796608
16:20 114131072 145027584 505458000 2804160
16:30 117556032 148438944 516472000 2810880
16:40 119440640 152107200 530502000 2778112
16:50 122959232 154380672 539360000 2790144
17:00 124928496 158689440 552207000 2800768
17:10 126891776 158847552 552512000 2812416
17:20 127855024 158224320 550382000 2809216
17:30 131414624 175564224 612382000 2820224
17:40 134469632 181457280 631190000 2832448
17:50 134728000 180476928 627692000 2834752
18:00 135065304 180194976 626369000 2837248
18:10 135048936 180101664 625861000 2839744
18:20 135131216 180012960 625274000 2840640
18:30 135184016 179980416 625136000 2840768
18:40 134980208 179988192 625164000 2844864
18:50 144367608 230463072 809997000 2841984
19:00 143008624 207547488 720874000 2845760
19:10 142775160 207562464 720707000 2849664
19:20 143572088 207624672 724979000 2858752
19:30 142798480 207610848 721979000 2863040
19:40 18940240 68677632 388814000 2844544
19:50 24506944 80140032 284238000 2876288
20:00 33779328 109852704 368152000 2896640
20:10 39744760 130029408 450371000 2927872
20:20 63298752 218945376 794323000 2962112
20:30 99855008 347968224 1220066000 3093824
20:40 133762112 431508096 1498229000 3230208
20:50 136770304 441884448 1540560000 3235072
21:00 143755744 462537504 1604335000 3251712
21:10 151189984 475643232 1652439000 3262528
21:20 157106400 480849408 1671900000 3251520
21:30 163313216 487090656 1690416000 3273728
21:40 169934336 498551040 1731005000 3275776
21:50 175835968 506492352 1761175000 3292672
22:00 182476800 516803040 1795376000 3299200
22:10 190756544 532979712 1856787000 3310016
22:20 197719808 539955936 1877669000 3319104
22:30 202812896 546294528 1910587000 3322944
22:40 209630080 552107520 1916452000 3333760
22:50 214403200 558634752 1941940000 3338688
23:00 220982784 567669024 1980128000 3340224
23:10 226360992 574801632 1999039000 3352768
23:20 232831104 583352640 2026452000 3333440
23:30 238333216 588961152 2049620000 3366656
23:40 245980416 600810912 2087720000 3379520
23:50 267191584 669404448 2325403000 3403520
00:00 267194048 668599200 2321551000 3405824 end 00:06 (I think)
00:10 267309152 668577888 2321454000 3408704
00:20 267289792 668578752 2321461000 3409600
00:30 267462272 668578752 2321461000 3411456
00:40 267234880 668578752 2321461000 3411840
00:50 267410880 668579616 2321468000 3416448
01:00 267192640 668595744 2321646000 3418048
01:10 267192640 668595744 2321646000 3418048
01:20 267192640 668595744 2321646000 3418048
01:30 267192640 668595744 2321646000 3418048
01:40 267192640 668595744 2321646000 3418048
Don't know if that's relevant, but I got another one of those:
Jul 4 12:50:51 ------------[ cut here ]------------
Jul 4 12:50:51 kernel BUG at
/media/data/mattems/src/linux-2.6-3.0.0~rc5/debian/build/source_amd64_none/fs/btrfs/inode.c:1583!
Jul 4 12:50:51 invalid opcode: 0000 [#1] SMP
Jul 4 12:50:51 CPU 0
Jul 4 12:50:51 Modules linked in: lm85 dme1737 hwmon_vid coretemp ipmi_si
ipmi_msghandler sha256_generic cryptd aes_x86_64 aes_generic cbc fuse dm_crypt
snd_pcm snd_timer snd sg soundcore i3200_edac snd_page_alloc sr_mod processor
tpm_tis i2c_i801 pl2303 pcspkr thermal_sys i2c_core tpm edac_core tpm_bios
cdrom usbserial container evdev psmouse button serio_raw ext4 mbcache jbd2
crc16 dm_mod nbd btrfs zlib_deflate crc32c libcrc32c ums_cypress sd_mod
crc_t10dif usb_storage uas uhci_hcd ahci libahci ehci_hcd libata e1000e usbcore
scsi_mod [last unloaded: i2c_dev]
Jul 4 12:50:51
Jul 4 12:50:51 Pid: 764, comm: btrfs-fixup-0 Not tainted 3.0.0-rc5-amd64 #1
empty empty/Tyan Tank GT20 B5211
Jul 4 12:50:51 RIP: 0010:[<ffffffffa00820f4>] [<ffffffffa00820f4>]
btrfs_writepage_fixup_worker+0xdb/0x120 [btrfs]
Jul 4 12:50:51 RSP: 0018:ffff880147ffdde0 EFLAGS: 00010246
Jul 4 12:50:51 RAX: 0000000000000000 RBX: ffffea0004648098 RCX:
ffffffffffffffff
Jul 4 12:50:51 RDX: 0000000000000000 RSI: 0000000005854000 RDI:
ffff8800073f18d0
Jul 4 12:50:51 RBP: 0000000005854000 R08: 0000000000000008 R09:
ffff880147ffdd98
Jul 4 12:50:51 R10: dead000000200200 R11: dead000000100100 R12:
ffff8800073f19a8
Jul 4 12:50:51 R13: 0000000000000000 R14: ffff880147ffddf8 R15:
0000000005854fff
Jul 4 12:50:51 FS: 0000000000000000(0000) GS:ffff88014fc00000(0000)
knlGS:0000000000000000
Jul 4 12:50:51 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 4 12:50:51 CR2: 00000000f75c6000 CR3: 000000014780f000 CR4:
00000000000006f0
Jul 4 12:50:51 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Jul 4 12:50:51 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Jul 4 12:50:51 Process btrfs-fixup-0 (pid: 764, threadinfo ffff880147ffc000,
task ffff880145d962c0)
Jul 4 12:50:51 Stack:
Jul 4 12:50:51 ffff8800073f1840 ffff8800040b9ec0 ffffffffffffffff
ffff8800c45ef5a8
Jul 4 12:50:51 0000000000000000 ffff880147c13440 ffff880147c13490
ffff880147c13458
Jul 4 12:50:51 ffff880147c13448 ffff880147c13480 ffff880147c13468
ffffffffa009d9fa
Jul 4 12:50:51 Call Trace:
Jul 4 12:50:51 [<ffffffffa009d9fa>] ? worker_loop+0x186/0x4a1 [btrfs]
Jul 4 12:50:51 [<ffffffff813369ca>] ? schedule+0x5ed/0x61a
Jul 4 12:50:51 [<ffffffffa009d874>] ? btrfs_queue_worker+0x24a/0x24a [btrfs]
Jul 4 12:50:51 [<ffffffffa009d874>] ? btrfs_queue_worker+0x24a/0x24a [btrfs]
Jul 4 12:50:51 [<ffffffff8105faed>] ? kthread+0x7a/0x82
Jul 4 12:50:51 [<ffffffff8133e524>] ? kernel_thread_helper+0x4/0x10
Jul 4 12:50:51 [<ffffffff8105fa73>] ? kthread_worker_fn+0x147/0x147
Jul 4 12:50:51 [<ffffffff8133e520>] ? gs_change+0x13/0x13
Jul 4 12:50:51 Code: 41 b8 50 00 00 00 4c 89 f1 e8 d5 3b 01 00 48 89 df e8 fb
3c 03 e1 ba 01 00 00 00 4c 89 ee 4c 89 e7 e8 ce 05 01 00 e9 4e ff ff ff <0f>
0beb fe 48 8b 3c 24 41 b8 50 00 00 00 4c 89 f1 4c 89 fa 48
Jul 4 12:50:51 RIP [<ffffffffa00820f4>]
btrfs_writepage_fixup_worker+0xdb/0x120 [btrfs]
Jul 4 12:50:51 RSP <ffff880147ffdde0>
Jul 4 12:50:51 ---[ end trace 3b143d6e02d6e847 ]---
Best regards,
Stephane
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
