If you are referring to the version of the Pvfs2 source then it is 2.8.2 Thanks Vish
On Tue, Jan 3, 2012 at 1:02 PM, Becky Ligon <[email protected]> wrote: > That is interesting. We have not tried to run the server on SSD, so there > may be differences in allocation between ssd and hard drives. We will have > to investigate. Can you tell me which version of the code you are using? > If you issue pvfs2-server --version, the version will be displayed. > > Thanks, > Becky > > On Tue, Jan 3, 2012 at 12:49 PM, Vishwanath Venkatesan < > [email protected]> wrote: > >> Hi, >> >> We have a pvfs2 filesystem over an SSD storage of 2TB. There are 2 pvfs2 >> servers mounted over two sections of the storage each viewing 1TB. There >> are 16 compute nodes which are pvfs2 clients. When I did a write of 65G >> from one compute node to the file system and watched the log there were >> some page allocation errors. Although the write did complete successfully I >> am suspecting whether this might pull down the performance of the PVFS2 >> filesystem. I have provided the trace, any insight from pvfs2 experts will >> be really helpful. >> >> The error trace looked like >> ######################################## >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355862] pvfs2-server: page >> allocation failure. order:0, mode:0x20 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355868] Pid: 24210, comm: >> pvfs2-server Not tainted 2.6.30-perfctr #8 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355871] Call Trace: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355873] <IRQ> >> [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355887] >> [<ffffffff802dc332>] alloc_pages_current+0x82/0xd0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355904] >> [<ffffffffa03c7a2f>] ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355908] >> [<ffffffff802126e0>] ? nommu_map_page+0x0/0xd0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355916] >> [<ffffffffa03c95d7>] ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355922] >> [<ffffffffa03c2144>] ipoib_poll+0xe4/0x1c0 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355927] >> [<ffffffff804dc4a7>] net_rx_action+0x117/0x1d0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355932] >> [<ffffffff8024fff4>] __do_softirq+0x84/0x210 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355935] >> [<ffffffff8020d0ac>] call_softirq+0x1c/0x30 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355937] >> [<ffffffff8020e84d>] do_softirq+0x3d/0x80 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355940] >> [<ffffffff8025027d>] irq_exit+0x8d/0x90 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355942] >> [<ffffffff8020e565>] do_IRQ+0x85/0xf0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355947] >> [<ffffffff8020c913>] ret_from_intr+0x0/0xa >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355948] <EOI> >> [<ffffffff802b9146>] ? shrink_page_list+0x686/0x820 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355955] >> [<ffffffff8020c90e>] ? common_interrupt+0xe/0x13 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355958] >> [<ffffffff802b98e8>] ? shrink_list+0x1f8/0x5d0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355961] >> [<ffffffff802ba240>] ? shrink_zone+0x240/0x360 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355966] >> [<ffffffff8026a477>] ? getnstimeofday+0x57/0xe0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355968] >> [<ffffffff802ba88e>] ? try_to_free_pages+0x27e/0x430 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355971] >> [<ffffffff802b8080>] ? isolate_pages_global+0x0/0x2a0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355975] >> [<ffffffff802b36ac>] ? __alloc_pages_internal+0x1fc/0x490 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355979] >> [<ffffffff802dc332>] ? alloc_pages_current+0x82/0xd0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355982] >> [<ffffffff802b01be>] ? __get_free_pages+0xe/0x80 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355985] >> [<ffffffff8024820d>] ? copy_process+0xbd/0x13d0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355988] >> [<ffffffff802495d0>] ? do_fork+0x80/0x400 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355991] >> [<ffffffff8020a9d3>] ? sys_clone+0x23/0x30 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355994] >> [<ffffffff8020c2d3>] ? stub_clone+0x13/0x20 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355998] >> [<ffffffff8020bf6b>] ? system_call_fastpath+0x16/0x1b >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356000] Mem-Info: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356002] Node 0 DMA per-cpu: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356005] CPU 0: hi: 0, >> btch: 1 usd: 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356007] CPU 1: hi: 0, >> btch: 1 usd: 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356008] Node 0 DMA32 per-cpu: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356011] CPU 0: hi: 186, >> btch: 31 usd: 75 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356013] CPU 1: hi: 186, >> btch: 31 usd: 54 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356017] Active_anon:2303 >> active_file:4218 inactive_anon:4108 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356018] >> inactive_file:464726 unevictable:0 dirty:48321 writeback:0 unstable:0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356019] free:2531 >> slab:21979 mapped:1458 pagetables:538 bounce:0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356021] Node 0 DMA >> free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB >> active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB >> pages_scanned:0 all_unreclaimable? no >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356027] lowmem_reserve[]: 0 >> 2003 2003 2003 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356030] Node 0 DMA32 >> free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB >> inactive_anon:16432kB active_file:16336kB inactive_file:1858752kB >> unevictable:0kB present:2051244kB pages_scanned:129 all_unreclaimable? no >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356035] lowmem_reserve[]: 0 >> 0 0 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356038] Node 0 DMA: 2*4kB >> 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB >> 1*4096kB = 8008kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356046] Node 0 DMA32: 0*4kB >> 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB >> 0*4096kB = 2056kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356054] 472673 total >> pagecache pages >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356055] 3606 pages in swap >> cache >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356057] Swap cache stats: >> add 1626590, delete 1622984, find 11678569/11742820 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356059] Free swap = >> 2073400kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356060] Total swap = >> 2095096kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368134] 524016 pages RAM >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368136] 9162 pages reserved >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368137] 469049 pages shared >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368139] 42601 pages >> non-shared >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368170] kswapd0: page >> allocation failure. order:0, mode:0x20 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368174] Pid: 24, comm: >> kswapd0 Not tainted 2.6.30-perfctr #8 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368175] Call Trace: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368177] <IRQ> >> [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368187] >> [<ffffffff802dc332>] alloc_pages_current+0x82/0xd0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368197] >> [<ffffffffa03c7a2f>] ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368202] >> [<ffffffff802126e0>] ? nommu_map_page+0x0/0xd0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368208] >> [<ffffffffa03c95d7>] ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368215] >> [<ffffffffa03c2144>] ipoib_poll+0xe4/0x1c0 [ib_ipoib] >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368219] >> [<ffffffff804dc4a7>] net_rx_action+0x117/0x1d0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368224] >> [<ffffffff8024fff4>] __do_softirq+0x84/0x210 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368227] >> [<ffffffff8020d0ac>] call_softirq+0x1c/0x30 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368229] >> [<ffffffff8020e84d>] do_softirq+0x3d/0x80 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368232] >> [<ffffffff8025027d>] irq_exit+0x8d/0x90 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368234] >> [<ffffffff8020e565>] do_IRQ+0x85/0xf0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368239] >> [<ffffffff8020c913>] ret_from_intr+0x0/0xa >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368240] <EOI> >> [<ffffffff805ab462>] ? thread_return+0x74/0x6e2 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368248] >> [<ffffffff805abae8>] ? schedule+0x18/0x40 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368251] >> [<ffffffff802bb1a9>] ? kswapd+0x769/0x780 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368254] >> [<ffffffff802b8080>] ? isolate_pages_global+0x0/0x2a0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368258] >> [<ffffffff80261b50>] ? autoremove_wake_function+0x0/0x40 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368261] >> [<ffffffff802baa40>] ? kswapd+0x0/0x780 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368263] >> [<ffffffff802baa40>] ? kswapd+0x0/0x780 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368266] >> [<ffffffff80261498>] ? kthread+0x58/0xa0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368269] >> [<ffffffff8020cfaa>] ? child_rip+0xa/0x20 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368272] >> [<ffffffff80261440>] ? kthread+0x0/0xa0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368275] >> [<ffffffff8020cfa0>] ? child_rip+0x0/0x20 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368276] Mem-Info: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368277] Node 0 DMA per-cpu: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368280] CPU 0: hi: 0, >> btch: 1 usd: 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368282] CPU 1: hi: 0, >> btch: 1 usd: 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368283] Node 0 DMA32 per-cpu: >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368286] CPU 0: hi: 186, >> btch: 31 usd: 75 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368288] CPU 1: hi: 186, >> btch: 31 usd: 182 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368292] Active_anon:2303 >> active_file:4218 inactive_anon:4108 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368293] >> inactive_file:464595 unevictable:0 dirty:48321 writeback:0 unstable:0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368294] free:2531 >> slab:21979 mapped:1458 pagetables:538 bounce:0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368296] Node 0 DMA >> free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB >> active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB >> pages_scanned:0 all_unreclaimable? no >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368301] lowmem_reserve[]: 0 >> 2003 2003 2003 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368304] Node 0 DMA32 >> free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB >> inactive_anon:16432kB active_file:16336kB inactive_file:1858228kB >> unevictable:0kB present:2051244kB pages_scanned:257 all_unreclaimable? no >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368310] lowmem_reserve[]: 0 >> 0 0 0 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368312] Node 0 DMA: 2*4kB >> 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB >> 1*4096kB = 8008kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368321] Node 0 DMA32: 0*4kB >> 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB >> 0*4096kB = 2056kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368329] 472549 total >> pagecache pages >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368330] 3606 pages in swap >> cache >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368332] Swap cache stats: >> add 1626590, delete 1622984, find 11678569/11742820 >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368334] Free swap = >> 2073400kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368335] Total swap = >> 2095096kB >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380288] 524016 pages RAM >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380290] 9162 pages reserved >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380291] 469017 pages shared >> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380292] 42601 pages >> non-shared >> >> ############################################################################### >> >> >> Thanks >> Vish >> >> >> >> >> _______________________________________________ >> Pvfs2-users mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> >> > > > -- > Becky Ligon > OrangeFS Support and Development > Omnibond Systems > Anderson, South Carolina > > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
