That is interesting. We have not tried to run the server on SSD, so there may be differences in allocation between ssd and hard drives. We will have to investigate. Can you tell me which version of the code you are using? If you issue pvfs2-server --version, the version will be displayed.
Thanks, Becky On Tue, Jan 3, 2012 at 12:49 PM, Vishwanath Venkatesan <[email protected]>wrote: > Hi, > > We have a pvfs2 filesystem over an SSD storage of 2TB. There are 2 pvfs2 > servers mounted over two sections of the storage each viewing 1TB. There > are 16 compute nodes which are pvfs2 clients. When I did a write of 65G > from one compute node to the file system and watched the log there were > some page allocation errors. Although the write did complete successfully I > am suspecting whether this might pull down the performance of the PVFS2 > filesystem. I have provided the trace, any insight from pvfs2 experts will > be really helpful. > > The error trace looked like > ######################################## > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355862] pvfs2-server: page > allocation failure. order:0, mode:0x20 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355868] Pid: 24210, comm: > pvfs2-server Not tainted 2.6.30-perfctr #8 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355871] Call Trace: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355873] <IRQ> > [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355887] [<ffffffff802dc332>] > alloc_pages_current+0x82/0xd0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355904] [<ffffffffa03c7a2f>] > ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355908] [<ffffffff802126e0>] > ? nommu_map_page+0x0/0xd0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355916] [<ffffffffa03c95d7>] > ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355922] [<ffffffffa03c2144>] > ipoib_poll+0xe4/0x1c0 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355927] [<ffffffff804dc4a7>] > net_rx_action+0x117/0x1d0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355932] [<ffffffff8024fff4>] > __do_softirq+0x84/0x210 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355935] [<ffffffff8020d0ac>] > call_softirq+0x1c/0x30 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355937] [<ffffffff8020e84d>] > do_softirq+0x3d/0x80 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355940] [<ffffffff8025027d>] > irq_exit+0x8d/0x90 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355942] [<ffffffff8020e565>] > do_IRQ+0x85/0xf0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355947] [<ffffffff8020c913>] > ret_from_intr+0x0/0xa > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355948] <EOI> > [<ffffffff802b9146>] ? shrink_page_list+0x686/0x820 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355955] [<ffffffff8020c90e>] > ? common_interrupt+0xe/0x13 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355958] [<ffffffff802b98e8>] > ? shrink_list+0x1f8/0x5d0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355961] [<ffffffff802ba240>] > ? shrink_zone+0x240/0x360 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355966] [<ffffffff8026a477>] > ? getnstimeofday+0x57/0xe0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355968] [<ffffffff802ba88e>] > ? try_to_free_pages+0x27e/0x430 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355971] [<ffffffff802b8080>] > ? isolate_pages_global+0x0/0x2a0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355975] [<ffffffff802b36ac>] > ? __alloc_pages_internal+0x1fc/0x490 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355979] [<ffffffff802dc332>] > ? alloc_pages_current+0x82/0xd0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355982] [<ffffffff802b01be>] > ? __get_free_pages+0xe/0x80 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355985] [<ffffffff8024820d>] > ? copy_process+0xbd/0x13d0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355988] [<ffffffff802495d0>] > ? do_fork+0x80/0x400 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355991] [<ffffffff8020a9d3>] > ? sys_clone+0x23/0x30 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355994] [<ffffffff8020c2d3>] > ? stub_clone+0x13/0x20 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.355998] [<ffffffff8020bf6b>] > ? system_call_fastpath+0x16/0x1b > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356000] Mem-Info: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356002] Node 0 DMA per-cpu: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356005] CPU 0: hi: 0, > btch: 1 usd: 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356007] CPU 1: hi: 0, > btch: 1 usd: 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356008] Node 0 DMA32 per-cpu: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356011] CPU 0: hi: 186, > btch: 31 usd: 75 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356013] CPU 1: hi: 186, > btch: 31 usd: 54 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356017] Active_anon:2303 > active_file:4218 inactive_anon:4108 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356018] inactive_file:464726 > unevictable:0 dirty:48321 writeback:0 unstable:0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356019] free:2531 slab:21979 > mapped:1458 pagetables:538 bounce:0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356021] Node 0 DMA > free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB > active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB > pages_scanned:0 all_unreclaimable? no > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356027] lowmem_reserve[]: 0 > 2003 2003 2003 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356030] Node 0 DMA32 > free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB > inactive_anon:16432kB active_file:16336kB inactive_file:1858752kB > unevictable:0kB present:2051244kB pages_scanned:129 all_unreclaimable? no > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356035] lowmem_reserve[]: 0 0 > 0 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356038] Node 0 DMA: 2*4kB > 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB > 1*4096kB = 8008kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356046] Node 0 DMA32: 0*4kB > 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB > 0*4096kB = 2056kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356054] 472673 total > pagecache pages > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356055] 3606 pages in swap > cache > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356057] Swap cache stats: add > 1626590, delete 1622984, find 11678569/11742820 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356059] Free swap = 2073400kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.356060] Total swap = 2095096kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368134] 524016 pages RAM > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368136] 9162 pages reserved > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368137] 469049 pages shared > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368139] 42601 pages non-shared > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368170] kswapd0: page > allocation failure. order:0, mode:0x20 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368174] Pid: 24, comm: > kswapd0 Not tainted 2.6.30-perfctr #8 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368175] Call Trace: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368177] <IRQ> > [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368187] [<ffffffff802dc332>] > alloc_pages_current+0x82/0xd0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368197] [<ffffffffa03c7a2f>] > ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368202] [<ffffffff802126e0>] > ? nommu_map_page+0x0/0xd0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368208] [<ffffffffa03c95d7>] > ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368215] [<ffffffffa03c2144>] > ipoib_poll+0xe4/0x1c0 [ib_ipoib] > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368219] [<ffffffff804dc4a7>] > net_rx_action+0x117/0x1d0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368224] [<ffffffff8024fff4>] > __do_softirq+0x84/0x210 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368227] [<ffffffff8020d0ac>] > call_softirq+0x1c/0x30 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368229] [<ffffffff8020e84d>] > do_softirq+0x3d/0x80 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368232] [<ffffffff8025027d>] > irq_exit+0x8d/0x90 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368234] [<ffffffff8020e565>] > do_IRQ+0x85/0xf0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368239] [<ffffffff8020c913>] > ret_from_intr+0x0/0xa > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368240] <EOI> > [<ffffffff805ab462>] ? thread_return+0x74/0x6e2 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368248] [<ffffffff805abae8>] > ? schedule+0x18/0x40 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368251] [<ffffffff802bb1a9>] > ? kswapd+0x769/0x780 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368254] [<ffffffff802b8080>] > ? isolate_pages_global+0x0/0x2a0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368258] [<ffffffff80261b50>] > ? autoremove_wake_function+0x0/0x40 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368261] [<ffffffff802baa40>] > ? kswapd+0x0/0x780 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368263] [<ffffffff802baa40>] > ? kswapd+0x0/0x780 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368266] [<ffffffff80261498>] > ? kthread+0x58/0xa0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368269] [<ffffffff8020cfaa>] > ? child_rip+0xa/0x20 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368272] [<ffffffff80261440>] > ? kthread+0x0/0xa0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368275] [<ffffffff8020cfa0>] > ? child_rip+0x0/0x20 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368276] Mem-Info: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368277] Node 0 DMA per-cpu: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368280] CPU 0: hi: 0, > btch: 1 usd: 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368282] CPU 1: hi: 0, > btch: 1 usd: 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368283] Node 0 DMA32 per-cpu: > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368286] CPU 0: hi: 186, > btch: 31 usd: 75 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368288] CPU 1: hi: 186, > btch: 31 usd: 182 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368292] Active_anon:2303 > active_file:4218 inactive_anon:4108 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368293] inactive_file:464595 > unevictable:0 dirty:48321 writeback:0 unstable:0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368294] free:2531 slab:21979 > mapped:1458 pagetables:538 bounce:0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368296] Node 0 DMA > free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB > active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB > pages_scanned:0 all_unreclaimable? no > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368301] lowmem_reserve[]: 0 > 2003 2003 2003 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368304] Node 0 DMA32 > free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB > inactive_anon:16432kB active_file:16336kB inactive_file:1858228kB > unevictable:0kB present:2051244kB pages_scanned:257 all_unreclaimable? no > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368310] lowmem_reserve[]: 0 0 > 0 0 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368312] Node 0 DMA: 2*4kB > 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB > 1*4096kB = 8008kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368321] Node 0 DMA32: 0*4kB > 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB > 0*4096kB = 2056kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368329] 472549 total > pagecache pages > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368330] 3606 pages in swap > cache > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368332] Swap cache stats: add > 1626590, delete 1622984, find 11678569/11742820 > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368334] Free swap = 2073400kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.368335] Total swap = 2095096kB > Dec 27 15:53:46 ioserver-02 kernel: [3026630.380288] 524016 pages RAM > Dec 27 15:53:46 ioserver-02 kernel: [3026630.380290] 9162 pages reserved > Dec 27 15:53:46 ioserver-02 kernel: [3026630.380291] 469017 pages shared > Dec 27 15:53:46 ioserver-02 kernel: [3026630.380292] 42601 pages non-shared > > ############################################################################### > > > Thanks > Vish > > > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
