Also, which kernel version are you using and which SSD card are you using. Becky
On Wed, Feb 15, 2012 at 10:18 AM, Becky Ligon <[email protected]> wrote: > > > On Wed, Feb 15, 2012 at 10:18 AM, Becky Ligon <[email protected]> wrote: > >> Vish: >> >> I have not figured out why you are getting this error. My co-worker, who >> has installed the server on a SSD, never saw this problem. I will give it >> a try on our machines and see what happens. Can you send me your OrangeFS >> configuration file and the version of OrangeFS that you are using? >> >> Thanks, >> Becky >> >> >> On Tue, Feb 14, 2012 at 4:45 PM, Vishwanath Venkatesan < >> [email protected]> wrote: >> >>> Hi Becky, >>> * >>> * >>> I had sent this email a long time ago. I had a question in this. Can you >>> tell me what this error means. It looks like an overflow to me. I mean any >>> insight on why the error could occur. >>> Please let me know. >>> >>> >>> >>> Thanks >>> Vish >>> On Tue, Jan 3, 2012 at 1:02 PM, Becky Ligon <[email protected]> wrote: >>> >>>> That is interesting. We have not tried to run the server on SSD, so >>>> there may be differences in allocation between ssd and hard drives. We >>>> will have to investigate. Can you tell me which version of the code you >>>> are using? If you issue pvfs2-server --version, the version will be >>>> displayed. >>>> >>>> Thanks, >>>> Becky >>>> >>>> On Tue, Jan 3, 2012 at 12:49 PM, Vishwanath Venkatesan < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> We have a pvfs2 filesystem over an SSD storage of 2TB. There are 2 >>>>> pvfs2 servers mounted over two sections of the storage each viewing 1TB. >>>>> There are 16 compute nodes which are pvfs2 clients. When I did a write of >>>>> 65G from one compute node to the file system and watched the log there >>>>> were >>>>> some page allocation errors. Although the write did complete successfully >>>>> I >>>>> am suspecting whether this might pull down the performance of the PVFS2 >>>>> filesystem. I have provided the trace, any insight from pvfs2 experts will >>>>> be really helpful. >>>>> >>>>> The error trace looked like >>>>> ######################################## >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355862] pvfs2-server: >>>>> page allocation failure. order:0, mode:0x20 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355868] Pid: 24210, comm: >>>>> pvfs2-server Not tainted 2.6.30-perfctr #8 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355871] Call Trace: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355873] <IRQ> >>>>> [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355887] >>>>> [<ffffffff802dc332>] alloc_pages_current+0x82/0xd0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355904] >>>>> [<ffffffffa03c7a2f>] ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355908] >>>>> [<ffffffff802126e0>] ? nommu_map_page+0x0/0xd0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355916] >>>>> [<ffffffffa03c95d7>] ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355922] >>>>> [<ffffffffa03c2144>] ipoib_poll+0xe4/0x1c0 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355927] >>>>> [<ffffffff804dc4a7>] net_rx_action+0x117/0x1d0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355932] >>>>> [<ffffffff8024fff4>] __do_softirq+0x84/0x210 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355935] >>>>> [<ffffffff8020d0ac>] call_softirq+0x1c/0x30 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355937] >>>>> [<ffffffff8020e84d>] do_softirq+0x3d/0x80 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355940] >>>>> [<ffffffff8025027d>] irq_exit+0x8d/0x90 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355942] >>>>> [<ffffffff8020e565>] do_IRQ+0x85/0xf0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355947] >>>>> [<ffffffff8020c913>] ret_from_intr+0x0/0xa >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355948] <EOI> >>>>> [<ffffffff802b9146>] ? shrink_page_list+0x686/0x820 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355955] >>>>> [<ffffffff8020c90e>] ? common_interrupt+0xe/0x13 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355958] >>>>> [<ffffffff802b98e8>] ? shrink_list+0x1f8/0x5d0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355961] >>>>> [<ffffffff802ba240>] ? shrink_zone+0x240/0x360 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355966] >>>>> [<ffffffff8026a477>] ? getnstimeofday+0x57/0xe0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355968] >>>>> [<ffffffff802ba88e>] ? try_to_free_pages+0x27e/0x430 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355971] >>>>> [<ffffffff802b8080>] ? isolate_pages_global+0x0/0x2a0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355975] >>>>> [<ffffffff802b36ac>] ? __alloc_pages_internal+0x1fc/0x490 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355979] >>>>> [<ffffffff802dc332>] ? alloc_pages_current+0x82/0xd0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355982] >>>>> [<ffffffff802b01be>] ? __get_free_pages+0xe/0x80 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355985] >>>>> [<ffffffff8024820d>] ? copy_process+0xbd/0x13d0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355988] >>>>> [<ffffffff802495d0>] ? do_fork+0x80/0x400 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355991] >>>>> [<ffffffff8020a9d3>] ? sys_clone+0x23/0x30 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355994] >>>>> [<ffffffff8020c2d3>] ? stub_clone+0x13/0x20 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.355998] >>>>> [<ffffffff8020bf6b>] ? system_call_fastpath+0x16/0x1b >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356000] Mem-Info: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356002] Node 0 DMA >>>>> per-cpu: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356005] CPU 0: hi: >>>>> 0, btch: 1 usd: 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356007] CPU 1: hi: >>>>> 0, btch: 1 usd: 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356008] Node 0 DMA32 >>>>> per-cpu: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356011] CPU 0: hi: >>>>> 186, btch: 31 usd: 75 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356013] CPU 1: hi: >>>>> 186, btch: 31 usd: 54 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356017] Active_anon:2303 >>>>> active_file:4218 inactive_anon:4108 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356018] >>>>> inactive_file:464726 unevictable:0 dirty:48321 writeback:0 unstable:0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356019] free:2531 >>>>> slab:21979 mapped:1458 pagetables:538 bounce:0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356021] Node 0 DMA >>>>> free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB >>>>> active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB >>>>> pages_scanned:0 all_unreclaimable? no >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356027] lowmem_reserve[]: >>>>> 0 2003 2003 2003 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356030] Node 0 DMA32 >>>>> free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB >>>>> inactive_anon:16432kB active_file:16336kB inactive_file:1858752kB >>>>> unevictable:0kB present:2051244kB pages_scanned:129 all_unreclaimable? no >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356035] lowmem_reserve[]: >>>>> 0 0 0 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356038] Node 0 DMA: 2*4kB >>>>> 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB >>>>> 1*4096kB = 8008kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356046] Node 0 DMA32: >>>>> 0*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB >>>>> 0*4096kB = 2056kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356054] 472673 total >>>>> pagecache pages >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356055] 3606 pages in >>>>> swap cache >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356057] Swap cache stats: >>>>> add 1626590, delete 1622984, find 11678569/11742820 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356059] Free swap = >>>>> 2073400kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.356060] Total swap = >>>>> 2095096kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368134] 524016 pages RAM >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368136] 9162 pages >>>>> reserved >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368137] 469049 pages >>>>> shared >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368139] 42601 pages >>>>> non-shared >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368170] kswapd0: page >>>>> allocation failure. order:0, mode:0x20 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368174] Pid: 24, comm: >>>>> kswapd0 Not tainted 2.6.30-perfctr #8 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368175] Call Trace: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368177] <IRQ> >>>>> [<ffffffff802b384d>] __alloc_pages_internal+0x39d/0x490 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368187] >>>>> [<ffffffff802dc332>] alloc_pages_current+0x82/0xd0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368197] >>>>> [<ffffffffa03c7a2f>] ipoib_cm_alloc_rx_skb+0xdf/0x460 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368202] >>>>> [<ffffffff802126e0>] ? nommu_map_page+0x0/0xd0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368208] >>>>> [<ffffffffa03c95d7>] ipoib_cm_handle_rx_wc+0x287/0x730 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368215] >>>>> [<ffffffffa03c2144>] ipoib_poll+0xe4/0x1c0 [ib_ipoib] >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368219] >>>>> [<ffffffff804dc4a7>] net_rx_action+0x117/0x1d0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368224] >>>>> [<ffffffff8024fff4>] __do_softirq+0x84/0x210 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368227] >>>>> [<ffffffff8020d0ac>] call_softirq+0x1c/0x30 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368229] >>>>> [<ffffffff8020e84d>] do_softirq+0x3d/0x80 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368232] >>>>> [<ffffffff8025027d>] irq_exit+0x8d/0x90 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368234] >>>>> [<ffffffff8020e565>] do_IRQ+0x85/0xf0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368239] >>>>> [<ffffffff8020c913>] ret_from_intr+0x0/0xa >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368240] <EOI> >>>>> [<ffffffff805ab462>] ? thread_return+0x74/0x6e2 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368248] >>>>> [<ffffffff805abae8>] ? schedule+0x18/0x40 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368251] >>>>> [<ffffffff802bb1a9>] ? kswapd+0x769/0x780 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368254] >>>>> [<ffffffff802b8080>] ? isolate_pages_global+0x0/0x2a0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368258] >>>>> [<ffffffff80261b50>] ? autoremove_wake_function+0x0/0x40 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368261] >>>>> [<ffffffff802baa40>] ? kswapd+0x0/0x780 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368263] >>>>> [<ffffffff802baa40>] ? kswapd+0x0/0x780 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368266] >>>>> [<ffffffff80261498>] ? kthread+0x58/0xa0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368269] >>>>> [<ffffffff8020cfaa>] ? child_rip+0xa/0x20 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368272] >>>>> [<ffffffff80261440>] ? kthread+0x0/0xa0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368275] >>>>> [<ffffffff8020cfa0>] ? child_rip+0x0/0x20 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368276] Mem-Info: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368277] Node 0 DMA >>>>> per-cpu: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368280] CPU 0: hi: >>>>> 0, btch: 1 usd: 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368282] CPU 1: hi: >>>>> 0, btch: 1 usd: 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368283] Node 0 DMA32 >>>>> per-cpu: >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368286] CPU 0: hi: >>>>> 186, btch: 31 usd: 75 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368288] CPU 1: hi: >>>>> 186, btch: 31 usd: 182 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368292] Active_anon:2303 >>>>> active_file:4218 inactive_anon:4108 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368293] >>>>> inactive_file:464595 unevictable:0 dirty:48321 writeback:0 unstable:0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368294] free:2531 >>>>> slab:21979 mapped:1458 pagetables:538 bounce:0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368296] Node 0 DMA >>>>> free:8008kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB >>>>> active_file:536kB inactive_file:152kB unevictable:0kB present:6744kB >>>>> pages_scanned:0 all_unreclaimable? no >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368301] lowmem_reserve[]: >>>>> 0 2003 2003 2003 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368304] Node 0 DMA32 >>>>> free:2116kB min:5716kB low:7144kB high:8572kB active_anon:9212kB >>>>> inactive_anon:16432kB active_file:16336kB inactive_file:1858228kB >>>>> unevictable:0kB present:2051244kB pages_scanned:257 all_unreclaimable? no >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368310] lowmem_reserve[]: >>>>> 0 0 0 0 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368312] Node 0 DMA: 2*4kB >>>>> 14*8kB 7*16kB 3*32kB 2*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 0*2048kB >>>>> 1*4096kB = 8008kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368321] Node 0 DMA32: >>>>> 0*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB >>>>> 0*4096kB = 2056kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368329] 472549 total >>>>> pagecache pages >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368330] 3606 pages in >>>>> swap cache >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368332] Swap cache stats: >>>>> add 1626590, delete 1622984, find 11678569/11742820 >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368334] Free swap = >>>>> 2073400kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.368335] Total swap = >>>>> 2095096kB >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380288] 524016 pages RAM >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380290] 9162 pages >>>>> reserved >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380291] 469017 pages >>>>> shared >>>>> Dec 27 15:53:46 ioserver-02 kernel: [3026630.380292] 42601 pages >>>>> non-shared >>>>> >>>>> ############################################################################### >>>>> >>>>> >>>>> Thanks >>>>> Vish >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Pvfs2-users mailing list >>>>> [email protected] >>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>> >>>>> >>>> >>>> >>>> -- >>>> Becky Ligon >>>> OrangeFS Support and Development >>>> Omnibond Systems >>>> Anderson, South Carolina >>>> >>>> >>>> >>> >> >> >> -- >> Becky Ligon >> OrangeFS Support and Development >> Omnibond Systems >> Anderson, South Carolina >> >> >> > > > -- > Becky Ligon > OrangeFS Support and Development > Omnibond Systems > Anderson, South Carolina > > > -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
