Thank you for the response. I suspect this is primarily an algorithm issue. The aufs directory read takes about 1 minute if the SSD directory has 10,000 files. It takes only 30 seconds if the SSD directory has 5000 files. It takes 1 second if the SSD directory has 100 files. Does aufs look at the read-only branch, once for every file in the writable branch? If so, why?
# time ls -U | wc -l 1391467 real 0m0.968s user 0m0.490s sys 0m0.490s > Please update aufs. It is VERY OLD (more than a year). Ok, I plan to update the entire Ubuntu distribution in a few weeks, so will try to update aufs then. If there is a serious issue (e.g. data corruption) let me know and I can discontinue using aufs until I have a chance to upgrade. So far it seems fine, except for the slow directory listing. > That is BAD. I am afraid you met kernel oops or something, and it is the > main reason of your slowness. > If you can, please show me the full message. I've attached the relevant part of /var/log/messages. Again, aufs speeds up if I remove files from the SSD directory. I have not rebooted. > Generally aufs in ubuntu doesn't work well because the aufs module in > ubuntu is too old. Good to know. Presumably this will get better with newer releases of Ubuntu. I guess I can also attempt to compile a recent aufs module for the existing kernel. > These large files are created on your first writable branch > (/data4/.aufs.xino). Is the capacity enough? Plenty of room. # df -h | grep -e "data[14]" -e aufs /dev/md0 932G 556G 376G 60% /data1 /dev/sdi 60G 16G 45G 26% /data4 none 60G 16G 45G 26% /aufs This is my second attempt at a union mount. My first attempt was mhddfs, which was a performance disaster. The hope is to find something that works perfectly.
Mar 30 19:55:01 mail-archive kernel: [2520586.263673] apache2: page allocation failure. order:8, mode:0x40d0 Mar 30 19:55:03 mail-archive kernel: [2520586.263681] Pid: 14589, comm: apache2 Not tainted 2.6.24-8-server #1 Mar 30 19:55:06 mail-archive kernel: [2520586.263682] Mar 30 19:55:06 mail-archive kernel: [2520586.263683] Call Trace: Mar 30 19:55:06 mail-archive kernel: [2520586.263705] [__alloc_pages+733/944] __alloc_pages+0x2dd/0x3b0 Mar 30 19:55:06 mail-archive kernel: [2520586.263710] [krealloc+52/208] krealloc+0x34/0xd0 Mar 30 19:55:06 mail-archive kernel: [2520586.263713] [__get_free_pages+14/64] __get_free_pages+0xe/0x40 Mar 30 19:55:06 mail-archive kernel: [2520586.263715] [krealloc+169/208] krealloc+0xa9/0xd0 Mar 30 19:55:06 mail-archive kernel: [2520586.263728] [_end+130900201/2129579936] :aufs:au_kzrealloc+0x39/0x60 Mar 30 19:55:06 mail-archive kernel: [2520586.263736] [_end+130877405/2129579936] :aufs:au_vdir_init+0x30d/0x4c0 Mar 30 19:55:06 mail-archive kernel: [2520586.263743] [_end+130869941/2129579936] :aufs:fbstart+0x5/0x10 Mar 30 19:55:06 mail-archive kernel: [2520586.263750] [_end+130863990/2129579936] :aufs:au_reval_and_lock_finfo+0x3a6/0x6e0 Mar 30 19:55:06 mail-archive kernel: [2520586.263756] [__down_write_nested+18/176] __down_write_nested+0x12/0xb0 Mar 30 19:55:06 mail-archive kernel: [2520586.263760] [filldir+0/240] filldir+0x0/0xf0 Mar 30 19:55:06 mail-archive kernel: [2520586.263767] [_end+130871026/2129579936] :aufs:aufs_readdir+0x82/0x130 Mar 30 19:55:06 mail-archive kernel: [2520586.263770] [filldir+0/240] filldir+0x0/0xf0 Mar 30 19:55:06 mail-archive kernel: [2520586.263773] [vfs_readdir+183/208] vfs_readdir+0xb7/0xd0 Mar 30 19:55:06 mail-archive kernel: [2520586.263776] [sys_getdents+150/224] sys_getdents+0x96/0xe0 Mar 30 19:55:06 mail-archive kernel: [2520586.263780] [system_call+126/131] system_call+0x7e/0x83 Mar 30 19:55:06 mail-archive kernel: [2520586.263784] Mar 30 19:55:06 mail-archive kernel: [2520586.263785] Mem-info: Mar 30 19:55:06 mail-archive kernel: [2520586.263787] Node 0 DMA per-cpu: Mar 30 19:55:06 mail-archive kernel: [2520586.263790] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263792] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263795] CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263797] CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263799] CPU 4: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263801] CPU 5: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263803] CPU 6: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263806] CPU 7: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263807] Node 0 DMA32 per-cpu: Mar 30 19:55:06 mail-archive kernel: [2520586.263810] CPU 0: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263812] CPU 1: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263814] CPU 2: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263816] CPU 3: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263818] CPU 4: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263821] CPU 5: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263823] CPU 6: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263825] CPU 7: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263827] Node 0 Normal per-cpu: Mar 30 19:55:06 mail-archive kernel: [2520586.263829] CPU 0: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263831] CPU 1: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263834] CPU 2: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263836] CPU 3: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263838] CPU 4: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263840] CPU 5: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263842] CPU 6: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263845] CPU 7: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 48 Mar 30 19:55:06 mail-archive kernel: [2520586.263848] Active:3217006 inactive:1930151 dirty:177205 writeback:2939 unstable:0 Mar 30 19:55:06 mail-archive kernel: [2520586.263849] free:2643290 slab:407071 mapped:5151 pagetables:21743 bounce:0 Mar 30 19:55:06 mail-archive kernel: [2520586.263850] Node 0 DMA free:10868kB min:4kB low:4kB high:4kB active:0kB inactive:0kB present:10404kB pages_scanned:0 all_unreclaimable? no Mar 30 19:55:06 mail-archive kernel: [2520586.263854] lowmem_reserve[]: 0 3255 32292 32292 Mar 30 19:55:06 mail-archive kernel: [2520586.263857] Node 0 DMA32 free:2946872kB min:2316kB low:2892kB high:3472kB active:0kB inactive:636kB present:3333344kB pages_scanned:0 all_unreclaimable? no Mar 30 19:55:06 mail-archive kernel: [2520586.263860] lowmem_reserve[]: 0 0 29037 29037 Mar 30 19:55:06 mail-archive kernel: [2520586.263863] Node 0 Normal free:7615420kB min:20676kB low:25844kB high:31012kB active:12868024kB inactive:7719968kB present:29734400kB pages_scanned:0 all_unreclaimable? no Mar 30 19:55:06 mail-archive kernel: [2520586.263868] lowmem_reserve[]: 0 0 0 0 Mar 30 19:55:06 mail-archive kernel: [2520586.263870] Node 0 DMA: 3*4kB 1*8kB 6*16kB 4*32kB 4*64kB 3*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10868kB Mar 30 19:55:06 mail-archive kernel: [2520586.263878] Node 0 DMA32: 73629*4kB 41446*8kB 34289*16kB 21382*32kB 9362*64kB 2889*128kB 426*256kB 18*512kB 1*1024kB 0*2048kB 0*4096kB = 2947188kB Mar 30 19:55:06 mail-archive kernel: [2520586.263886] Node 0 Normal: 919566*4kB 325031*8kB 52333*16kB 10096*32kB 2281*64kB 205*128kB 15*256kB 3*512kB 0*1024kB 0*2048kB 0*4096kB = 7616512kB Mar 30 19:55:06 mail-archive kernel: [2520586.263895] Swap cache: add 2273087, delete 2165179, find 12764885/12976223, race 0+194 Mar 30 19:55:06 mail-archive kernel: [2520586.263897] Free swap = 11852972kB Mar 30 19:55:06 mail-archive kernel: [2520586.263898] Total swap = 12699340kB Mar 30 19:55:06 mail-archive kernel: [2520586.263899] Free swap: 11852972kB Mar 30 19:55:06 mail-archive kernel: [2520586.459271] 8585216 pages of RAM Mar 30 19:55:06 mail-archive kernel: [2520586.459278] 332243 reserved pages Mar 30 19:55:06 mail-archive kernel: [2520586.459279] 1364830 pages shared Mar 30 19:55:06 mail-archive kernel: [2520586.459280] 107765 pages swap cached Mar 30 19:56:15 mail-archive kernel: [2520659.915075] apache2: page allocation failure. order:8, mode:0x40d0 Mar 30 19:56:15 mail-archive kernel: [2520659.915084] Pid: 20135, comm: apache2 Not tainted 2.6.24-8-server #1 Mar 30 19:56:15 mail-archive kernel: [2520659.915089] Mar 30 19:56:15 mail-archive kernel: [2520659.915089] Call Trace: Mar 30 19:56:15 mail-archive kernel: [2520659.915129] [__alloc_pages+733/944] __alloc_pages+0x2dd/0x3b0 Mar 30 19:56:15 mail-archive kernel: [2520659.915139] [__get_free_pages+14/64] __get_free_pages+0xe/0x40 Mar 30 19:56:15 mail-archive kernel: [2520659.915145] [krealloc+169/208] krealloc+0xa9/0xd0 Mar 30 19:56:15 mail-archive kernel: [2520659.915168] [_end+130900201/2129579936] :aufs:au_kzrealloc+0x39/0x60 Mar 30 19:56:15 mail-archive kernel: [2520659.915178] [_end+130877405/2129579936] :aufs:au_vdir_init+0x30d/0x4c0 Mar 30 19:56:15 mail-archive kernel: [2520659.915187] [_end+130869941/2129579936] :aufs:fbstart+0x5/0x10 Mar 30 19:56:15 mail-archive kernel: [2520659.915194] [_end+130863990/2129579936] :aufs:au_reval_and_lock_finfo+0x3a6/0x6e0 Mar 30 19:56:15 mail-archive kernel: [2520659.915201] [__down_write_nested+18/176] __down_write_nested+0x12/0xb0 Mar 30 19:56:15 mail-archive kernel: [2520659.915206] [filldir+0/240] filldir+0x0/0xf0 Mar 30 19:56:15 mail-archive kernel: [2520659.915214] [_end+130871026/2129579936] :aufs:aufs_readdir+0x82/0x130 Mar 30 19:56:15 mail-archive kernel: [2520659.915218] [filldir+0/240] filldir+0x0/0xf0 Mar 30 19:56:15 mail-archive kernel: [2520659.915222] [vfs_readdir+183/208] vfs_readdir+0xb7/0xd0 Mar 30 19:56:15 mail-archive kernel: [2520659.915226] [sys_getdents+150/224] sys_getdents+0x96/0xe0 Mar 30 19:56:15 mail-archive kernel: [2520659.915231] [system_call+126/131] system_call+0x7e/0x83 Mar 30 19:56:15 mail-archive kernel: [2520659.915235] Mar 30 19:56:15 mail-archive kernel: [2520659.915237] Mem-info: Mar 30 19:56:15 mail-archive kernel: [2520659.915240] Node 0 DMA per-cpu: Mar 30 19:56:15 mail-archive kernel: [2520659.915243] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915246] CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915249] CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915252] CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915255] CPU 4: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915258] CPU 5: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915261] CPU 6: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915264] CPU 7: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915267] Node 0 DMA32 per-cpu: Mar 30 19:56:15 mail-archive kernel: [2520659.915270] CPU 0: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915273] CPU 1: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915276] CPU 2: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915279] CPU 3: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915282] CPU 4: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915285] CPU 5: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915288] CPU 6: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915291] CPU 7: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915293] Node 0 Normal per-cpu: Mar 30 19:56:15 mail-archive kernel: [2520659.915297] CPU 0: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915301] CPU 1: Hot: hi: 186, btch: 31 usd: 7 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915304] CPU 2: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915307] CPU 3: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915310] CPU 4: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915314] CPU 5: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915317] CPU 6: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915320] CPU 7: Hot: hi: 186, btch: 31 usd: 0 Cold: hi: 62, btch: 15 usd: 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915324] Active:2942673 inactive:2114883 dirty:171613 writeback:53 unstable:0 Mar 30 19:56:15 mail-archive kernel: [2520659.915325] free:2669158 slab:467900 mapped:5287 pagetables:22633 bounce:0 Mar 30 19:56:15 mail-archive kernel: [2520659.915327] Node 0 DMA free:10868kB min:4kB low:4kB high:4kB active:0kB inactive:0kB present:10404kB pages_scanned:0 all_unreclaimable? no Mar 30 19:56:15 mail-archive kernel: [2520659.915332] lowmem_reserve[]: 0 3255 32292 32292 Mar 30 19:56:15 mail-archive kernel: [2520659.915337] Node 0 DMA32 free:2948084kB min:2316kB low:2892kB high:3472kB active:0kB inactive:596kB present:3333344kB pages_scanned:0 all_unreclaimable? no Mar 30 19:56:15 mail-archive kernel: [2520659.915341] lowmem_reserve[]: 0 0 29037 29037 Mar 30 19:56:15 mail-archive kernel: [2520659.915345] Node 0 Normal free:7717680kB min:20676kB low:25844kB high:31012kB active:11770692kB inactive:8458936kB present:29734400kB pages_scanned:0 all_unreclaimable? no Mar 30 19:56:15 mail-archive kernel: [2520659.915350] lowmem_reserve[]: 0 0 0 0 Mar 30 19:56:15 mail-archive kernel: [2520659.915355] Node 0 DMA: 3*4kB 1*8kB 6*16kB 4*32kB 4*64kB 3*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 1*4096kB = 10868kB Mar 30 19:56:15 mail-archive kernel: [2520659.915367] Node 0 DMA32: 73436*4kB 41291*8kB 34274*16kB 21384*32kB 9374*64kB 2906*128kB 428*256kB 18*512kB 1*1024kB 0*2048kB 0*4096kB = 2948456kB Mar 30 19:56:15 mail-archive kernel: [2520659.915383] Node 0 Normal: 692025*4kB 346657*8kB 69760*16kB 19545*32kB 5185*64kB 568*128kB 69*256kB 23*512kB 2*1024kB 0*2048kB 0*4096kB = 7718988kB Mar 30 19:56:15 mail-archive kernel: [2520659.915399] Swap cache: add 2295506, delete 2175004, find 12785325/12999962, race 0+246 Mar 30 19:56:15 mail-archive kernel: [2520659.915401] Free swap = 11832788kB Mar 30 19:56:15 mail-archive kernel: [2520659.915401] Total swap = 12699340kB Mar 30 19:56:15 mail-archive kernel: [2520659.915402] Free swap: 11832788kB Mar 30 19:56:15 mail-archive kernel: [2520660.110808] 8585216 pages of RAM Mar 30 19:56:15 mail-archive kernel: [2520660.110815] 332243 reserved pages Mar 30 19:56:15 mail-archive kernel: [2520660.110816] 1355196 pages shared Mar 30 19:56:15 mail-archive kernel: [2520660.110817] 120508 pages swap cached k
------------------------------------------------------------------------------
