Re: vmur and punching linux kernel images
Hello Pat, On Tue, 2009-12-22 at 21:14 -0600, Patrick Spinler wrote: I was just messing about with the vmur utility from the s390 tools, and I was wondering: has anyone attempted to punch a kernel, initrd + param file directly from (a suitably privileged) linux guest into another guests' reader? It'd be a nifty provisioning trick: generate the appropriate param file to do a kickstart install, then punch all the above to a new linux guest and 'vmcp xautolog linuxguest ipl 0C' it. Do all this directly from the NFS server I keep the OS images on. I'm curious because it doesn't look like vmcp allows me to specify a fixed record length - I can specify separator and padding bytes with the vmur punch --blocked option, but that doesn't really sound like what's required here. I'll continue to play around with this, but does anyone have any thoughts? This should work with the vmur tool. The target reader has to be empty and you have to punch the files in the correct order: # vmur pun -r -u guest image # vmur pun -r -u guest parmfile # vmur pun -r -u guest initrd We described the (local) procedure in the Device Driver Book (Chapter: vmur - Work with z/VM spool file queues): http://public.dhe.ibm.com/software/dw/linux390/docu/lk32dd04.pdf Michael -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
SLES10 - Oracle/Memory Issues (oom-killer)
We've noticed some repetitive error messages in regards to a memory issue on one of our SLES 10 SP2. We notice the error below every morning around the same time, with multiple oom-kills of oracle, perl, etc. Anyone have any thoughts as to why this might be happening with the amount of memory this guest has? If you have any thoughts or need anymore information, I would certainly appreciate it. #MEMINFO (For a reference) MemTotal: 5138052 kB MemFree:144308 kB Buffers:121768 kB Cached:3717360 kB SwapCached: 46352 kB Active:2849880 kB Inactive: 1590368 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 5138052 kB LowFree:144308 kB SwapTotal: 2352736 kB SwapFree: 2116048 kB Dirty: 376 kB Writeback: 0 kB AnonPages: 584892 kB Mapped:2159592 kB Slab: 172056 kB CommitLimit: 4921760 kB Committed_AS: 5506584 kB PageTables: 334304 kB VmallocTotal: 4289716224 kB VmallocUsed: 5024 kB VmallocChunk: 4289711032 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB #ERROR MESSAGE (excerpt from /var/log/warn, this error is repeated over and over, killing several different processes within a couple minutes) Dec 27 04:37:33 *SERVER NAME* kernel: oracle invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Dec 27 04:37:34 *SERVER NAME* kernel: ccd13918 c54c2cf1 8142d938 0061a350 Dec 27 04:37:34 *SERVER NAME* kernel:61104620 00105b52 Dec 27 04:37:34 *SERVER NAME* kernel: 006009c0 0418 Dec 27 04:37:34 *SERVER NAME* kernel:0001 0008 000e ccd139c0 Dec 27 04:37:34 *SERVER NAME* kernel:004ad158 00105b52 ccd13948 ccd13988 Dec 27 04:37:34 *SERVER NAME* kernel: Call Trace: Dec 27 04:37:34 *SERVER NAME* kernel: ([00105b6e] dump_stack+0x2aa/0x364) Dec 27 04:37:34 *SERVER NAME* kernel: [001c3994] out_of_memory+0x3b8/0x934 Dec 27 04:37:34 *SERVER NAME* kernel: [001c7216] __alloc_pages+0x29a/0x370 Dec 27 04:37:34 *SERVER NAME* kernel: [001ce582] do_page_cache_readahead+0x156/0x3a8 Dec 27 04:37:34 *SERVER NAME* kernel: [001bd55c] filemap_nopage+0x210/0xa4c Dec 27 04:37:34 *SERVER NAME* kernel: [001e18be] __handle_mm_fault+0x282/0x114c Dec 27 04:37:34 *SERVER NAME* kernel: [00102080] do_dat_exception+0x584/0x858 Dec 27 04:37:34 *SERVER NAME* kernel: [00115d16] sysc_return+0x0/0x10 Dec 27 04:37:34 *SERVER NAME* kernel: [8061041c] 0x8061041c Dec 27 04:37:34 *SERVER NAME* kernel: Dec 27 04:37:34 *SERVER NAME* kernel: Mem-info: Dec 27 04:37:34 *SERVER NAME* kernel: DMA per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 58 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 156 Cold: hi: 62, btch: 15 usd: 27 Dec 27 04:37:34 *SERVER NAME* kernel: Normal per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 56 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 14 Dec 27 04:37:34 *SERVER NAME* kernel: Free pages: 21488kB (0kB HighMem) Dec 27 04:37:34 *SERVER NAME* kernel: Active:832717 inactive:260329 dirty:0 writeback:0 unstable:0 free:5372 slab:14827 mapped:650 pagetables:14672 2 Dec 27 04:37:34 *SERVER NAME* kernel: DMA free:16016kB min:3660kB low:4572kB high:5488kB active:807232kB inactive:896400kB present:2097152kB pages_ scanned:3937815 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel: lowmem_reserve[]: 0 0 3072 3072 Dec 27 04:37:34 *SERVER NAME* kernel: Normal free:5472kB min:5492kB low:6864kB high:8236kB active:2523636kB inactive:144916kB present:3145728kB pag es_scanned:7066898 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel: lowmem_reserve[]: 0 0 0 0 Dec 27 04:37:34 *SERVER NAME* kernel: DMA: 1260*4kB 1258*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 16016kB Dec 27 04:37:34 *SERVER NAME* kernel: Normal: 10*4kB 507*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5472kB Dec 27 04:37:34 *SERVER NAME* kernel: Swap cache: add 31963696, delete 31964356, find 17660088/21209122, race 8+295 Dec 27 04:37:34 *SERVER NAME* kernel: Free swap = 0kB Dec 27 04:37:35 *SERVER NAME* kernel: Total swap = 2352736kB Dec 27 04:37:35 *SERVER NAME* kernel: Free swap:0kB Dec 27 04:37:35 *SERVER NAME* kernel: 1310720 pages of RAM Dec 27 04:37:35 *SERVER NAME* kernel: 26207 reserved pages Dec 27 04:37:35 *SERVER NAME* kernel: 66904 pages shared Dec 27 04:37:35 *SERVER NAME*
Re: SLES10 - Oracle/Memory Issues (oom-killer)
My first guess, is that you are out of swap space. Now why, is a different story. So, either increase the guest machine size for Oracle (do a q v stor first to see if what you think it should be, is actually what it is G), or add more swap space. I think there are some conditions where insufficient virtual memory can also lead to OOM. Something that needs lots of memory, and can't tolerate it being swapped. In that case, increase the guest machine size. Sometimes, I've cause that type of problem, by editing a large log file. Editors like joe, puts everything in memory. Got a 500 MB log file? No problemuntil you run out of swap space. Tom Duerbusch THD Consulting Also running with Oracle 10g R2 (4 copies) Rodery, Floyd A Mr CIV US DISA CDB12 floyd.rod...@csd.disa.mil 12/30/2009 4:48 PM We've noticed some repetitive error messages in regards to a memory issue on one of our SLES 10 SP2. We notice the error below every morning around the same time, with multiple oom-kills of oracle, perl, etc. Anyone have any thoughts as to why this might be happening with the amount of memory this guest has? If you have any thoughts or need anymore information, I would certainly appreciate it. #MEMINFO (For a reference) MemTotal: 5138052 kB MemFree:144308 kB Buffers:121768 kB Cached:3717360 kB SwapCached: 46352 kB Active:2849880 kB Inactive: 1590368 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 5138052 kB LowFree:144308 kB SwapTotal: 2352736 kB SwapFree: 2116048 kB Dirty: 376 kB Writeback: 0 kB AnonPages: 584892 kB Mapped:2159592 kB Slab: 172056 kB CommitLimit: 4921760 kB Committed_AS: 5506584 kB PageTables: 334304 kB VmallocTotal: 4289716224 kB VmallocUsed: 5024 kB VmallocChunk: 4289711032 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB #ERROR MESSAGE (excerpt from /var/log/warn, this error is repeated over and over, killing several different processes within a couple minutes) Dec 27 04:37:33 *SERVER NAME* kernel: oracle invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Dec 27 04:37:34 *SERVER NAME* kernel: ccd13918 c54c2cf1 8142d938 0061a350 Dec 27 04:37:34 *SERVER NAME* kernel:61104620 00105b52 Dec 27 04:37:34 *SERVER NAME* kernel: 006009c0 0418 Dec 27 04:37:34 *SERVER NAME* kernel:0001 0008 000e ccd139c0 Dec 27 04:37:34 *SERVER NAME* kernel:004ad158 00105b52 ccd13948 ccd13988 Dec 27 04:37:34 *SERVER NAME* kernel: Call Trace: Dec 27 04:37:34 *SERVER NAME* kernel: ([00105b6e] dump_stack+0x2aa/0x364) Dec 27 04:37:34 *SERVER NAME* kernel: [001c3994] out_of_memory+0x3b8/0x934 Dec 27 04:37:34 *SERVER NAME* kernel: [001c7216] __alloc_pages+0x29a/0x370 Dec 27 04:37:34 *SERVER NAME* kernel: [001ce582] do_page_cache_readahead+0x156/0x3a8 Dec 27 04:37:34 *SERVER NAME* kernel: [001bd55c] filemap_nopage+0x210/0xa4c Dec 27 04:37:34 *SERVER NAME* kernel: [001e18be] __handle_mm_fault+0x282/0x114c Dec 27 04:37:34 *SERVER NAME* kernel: [00102080] do_dat_exception+0x584/0x858 Dec 27 04:37:34 *SERVER NAME* kernel: [00115d16] sysc_return+0x0/0x10 Dec 27 04:37:34 *SERVER NAME* kernel: [8061041c] 0x8061041c Dec 27 04:37:34 *SERVER NAME* kernel: Dec 27 04:37:34 *SERVER NAME* kernel: Mem-info: Dec 27 04:37:34 *SERVER NAME* kernel: DMA per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 58 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 156 Cold: hi: 62, btch: 15 usd: 27 Dec 27 04:37:34 *SERVER NAME* kernel: Normal per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 56 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 14 Dec 27 04:37:34 *SERVER NAME* kernel: Free pages: 21488kB (0kB HighMem) Dec 27 04:37:34 *SERVER NAME* kernel: Active:832717 inactive:260329 dirty:0 writeback:0 unstable:0 free:5372 slab:14827 mapped:650 pagetables:14672 2 Dec 27 04:37:34 *SERVER NAME* kernel: DMA free:16016kB min:3660kB low:4572kB high:5488kB active:807232kB inactive:896400kB present:2097152kB pages_ scanned:3937815 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel: lowmem_reserve[]: 0 0 3072 3072 Dec 27 04:37:34 *SERVER NAME* kernel: Normal free:5472kB min:5492kB low:6864kB high:8236kB active:2523636kB inactive:144916kB present:3145728kB pag es_scanned:7066898 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel:
Re: SLES10 - Oracle/Memory Issues (oom-killer)
On Wed, Dec 30, 2009 at 11:48 PM, Rodery, Floyd A Mr CIV US DISA CDB12 floyd.rod...@csd.disa.mil wrote: We've noticed some repetitive error messages in regards to a memory issue on one of our SLES 10 SP2. We notice the error below every morning around the same time, with multiple oom-kills of oracle, perl, etc. Anyone have any thoughts as to why this might be happening with the amount of memory this guest has? If you have any thoughts or need anymore information, I would certainly appreciate it. So you filled memory and then ran out of swap space? Your first check would be to see whether SGA and PGA fit comfortably in the guest without swapping. If you're having swappiness still to the default of 60, I would recommend to get that down to 0 (certainly if you don't run Oracle on raw devices). Your ESAUCD2 report would show whether the page cache was far more than the SGA (happens with high swappiness) and whether configured PGA size could be handled. When you believe that SGA and PGA would fit, then you may be bitten by a bug in Oracle where it forgets about PGA setting and allocates far more. I can dig up the reference for that... Rob -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: SLES10 - Oracle/Memory Issues (oom-killer)
On 12/30/2009 at 5:48 PM, Rodery, Floyd A Mr CIV US DISA CDB12 floyd.rod...@csd.disa.mil wrote: We've noticed some repetitive error messages in regards to a memory issue on one of our SLES 10 SP2. We notice the error below every morning around the same time, with multiple oom-kills of oracle, perl, etc. Anyone have any thoughts as to why this might be happening with the amount of memory this guest has? If you have any thoughts or need anymore information, I would certainly appreciate it. -snip- Dec 27 04:37:34 *SERVER NAME* kernel: Mem-info: Dec 27 04:37:34 *SERVER NAME* kernel: DMA per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 58 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 156 Cold: hi: 62, btch: 15 usd: 27 Dec 27 04:37:34 *SERVER NAME* kernel: Normal per-cpu: Dec 27 04:37:34 *SERVER NAME* kernel: CPU0: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 56 Dec 27 04:37:34 *SERVER NAME* kernel: CPU1: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 14 Dec 27 04:37:34 *SERVER NAME* kernel: Free pages: 21488kB (0kB HighMem) Dec 27 04:37:34 *SERVER NAME* kernel: Active:832717 inactive:260329 dirty:0 writeback:0 unstable:0 free:5372 slab:14827 mapped:650 pagetables:14672 2 Dec 27 04:37:34 *SERVER NAME* kernel: DMA free:16016kB min:3660kB low:4572kB high:5488kB active:807232kB inactive:896400kB present:2097152kB pages_ scanned:3937815 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel: lowmem_reserve[]: 0 0 3072 3072 Dec 27 04:37:34 *SERVER NAME* kernel: Normal free:5472kB min:5492kB low:6864kB high:8236kB active:2523636kB inactive:144916kB present:3145728kB pag es_scanned:7066898 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* kernel: lowmem_reserve[]: 0 0 0 0 Dec 27 04:37:34 *SERVER NAME* kernel: DMA: 1260*4kB 1258*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 16016kB Dec 27 04:37:34 *SERVER NAME* kernel: Normal: 10*4kB 507*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5472kB Dec 27 04:37:34 *SERVER NAME* kernel: Swap cache: add 31963696, delete 31964356, find 17660088/21209122, race 8+295 Dec 27 04:37:34 *SERVER NAME* kernel: Free swap = 0kB Dec 27 04:37:35 *SERVER NAME* kernel: Total swap = 2352736kB Dec 27 04:37:35 *SERVER NAME* kernel: Free swap:0kB Dec 27 04:37:35 *SERVER NAME* kernel: 1310720 pages of RAM Dec 27 04:37:35 *SERVER NAME* kernel: 26207 reserved pages Dec 27 04:37:35 *SERVER NAME* kernel: 66904 pages shared Dec 27 04:37:35 *SERVER NAME* kernel: 352 pages swap cached Dec 27 04:37:35 *SERVER NAME* kernel: Out of Memory: Kill process 26578 (oracle) score 48132 and children. Dec 27 04:37:35 *SERVER NAME* kernel: Out of memory: Killed process 26578 (oracle). This says that at the time OOM was invoked, you had no free paging slots (Free swap = 0kB), no free memory (0kB HighMem), and no memory that could be reclaimed (all_unreclaimable? yes). What kicks off at that time of day? Some sort of Oracle housekeeping processes? Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: SLES10 - Oracle/Memory Issues (oom-killer)
An oldie, but I liked it: http://lkml.org/lkml/2004/9/23/311 Shane ... -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Rejected posting to LINUX-390@VM.MARIST.EDU
Rob van der Heij wrote: On Thu, Dec 24, 2009 at 12:38 AM, John Summerfield deb...@herakles.homelinux.org wrote: I had it happen a few days ago, and I've seen in in the past. I assumed that the listserv got confused, or someone restored a mail queue that had already been processed, or something. Yes, it happens a few days after the post. It's not uncommon for MTA's to try delivery for a few days before giving up and apparently return the item to the sender. I fear we need Harry or Martha to identify which subscriber does this... Rob It's recurred, and I have some evidence. Email from me left here: Received: from js.id.au (dsl-58-6-192-22.wa.westnet.com.au [58.6.192.22]) by smtp-inbound4.marist.edu with ESMTP id CA9RyR7qUlL9v5f5 for LINUX-390@VM.MARIST.EDU; Mon, 28 Dec 2009 18:13:22 -0500 (EST) That's the first Received: header verifiable outside my site. It spent a few minutes bouncing around Marist and friends: Return-Path: Received: from MARMAIL2 (NJE origin smtp...@marist) by VM.MARIST.EDU (LMail V1.2b/1.8b) with BSMTP id 4604; Tue, 29 Dec 2009 18:18:22 -0500 Received: from smtp-inbound5.marist.edu [148.100.49.28] by VMMAIL2.MARIST.EDU (IBM VM SMTP Level 540) via TCP with ESMTP ; Tue, 29 Dec 2009 Then it got wrapped up and returned. This is the first header from the message returning it: Received: from VM.MARIST.EDU (NJE origin lists...@marist) by VM.MARIST.EDU (LMail V1.2b/1.8b) with BSMTP id 4608; Tue, 29 Dec 2009 18:18:25 -0500 and arrived here: Received: from smtp-outbound3.marist.edu (smtp-outbound3.marist.edu [148.100.49.24]) by ns.demo.lan (Postfix) with ESMTP id A2E4AC2C016 for deb...@herakles Presumably folk at Marist can correlate this to their logs. Note that, while most timestamps are in times local to Marist, mine are not. -- Cheers John -- spambait 1...@coco.merseine.nu z1...@coco.merseine.nu -- Advice http://webfoot.com/advice/email.top.php http://www.catb.org/~esr/faqs/smart-questions.html http://support.microsoft.com/kb/555375 You cannot reply off-list:-) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Senior WebSphere Consultant
Jose Munoz wrote: Hi, I still working on WebSphere Application Server for z/OS and Windows platforms as Senior WebSphere Consultant since 9 years ago, my project finish January 2010 and I will be available February, 1st 2010. I have experience on zSeries for more than 25 years. Jose, please do not interpret this is as critical of you. I don't recall the community attitude to I want a job posts, but this is not a suitable forum for seeking employees. Posting job-vacant ads here can cause grief, including - so others have said - termination to readers whose employers might suppose they're looking for a new job. The focus of this community is using Linux on IBM mainframes. Contributing here might work towards establishing a reputation, and you could get some ideas about where you'd like to work in the future, but I'd hesitate to take it further than that. -- Cheers John -- spambait 1...@coco.merseine.nu z1...@coco.merseine.nu -- Advice http://webfoot.com/advice/email.top.php http://www.catb.org/~esr/faqs/smart-questions.html http://support.microsoft.com/kb/555375 You cannot reply off-list:-) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: Moving On: Cross-posted to linux and vm lists
Phil best of luck in your new role and look forward to your posts David Kreuter Original Message Subject: Moving On: Cross-posted to linux and vm lists From: Phil Tully tull...@optonline.net Date: Tue, December 29, 2009 8:12 pm To: LINUX-390@VM.MARIST.EDU To All, As of January 1, 2010, I will no longer work for IBM. The past 5 + years have been a rewarding experience for me, but I have found an opportunity that I could not ignore. The new situation will be better for me professionally, and personally. I will continue to work with VM and Linux on Z and will continue to be a member of both listservs. With regards to all the customers I have interacted with , Thank You. I hope you learned half as much as I did. Phil Tully -- 'in media stat virtus' Virtue's in the middle -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
