Mark, Currently looking into that, some housekeeping may be contributing. On a bigger scale, obviously some adjustments/tuning need to be made to allow the oracle app and housekeeping tasks to operate within acceptable server configurations.
#Floyd Rodery DoD - DISA - DECC MECH CDB 12 Linux on System z 717.605.8639 | 430.8639 [email protected] -----Original Message----- From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Mark Post Sent: Wednesday, December 30, 2009 18:10 To: [email protected] Subject: Re: SLES10 - Oracle/Memory Issues (oom-killer) >>> On 12/30/2009 at 5:48 PM, "Rodery, Floyd A Mr CIV US DISA CDB12" <[email protected]> wrote: > We've noticed some repetitive error messages in regards to a memory > issue on one of our SLES 10 SP2. We notice the error below every > morning around the same time, with multiple oom-kills of oracle, perl, > etc. Anyone have any thoughts as to why this might be happening with > the amount of memory this guest has? If you have any thoughts or need > anymore information, I would certainly appreciate it. -snip- > Dec 27 04:37:34 *SERVER NAME* kernel: Mem-info: > Dec 27 04:37:34 *SERVER NAME* kernel: DMA per-cpu: > Dec 27 04:37:34 *SERVER NAME* kernel: CPU 0: Hot: hi: 186, btch: > 31 usd: 159 Cold: hi: 62, btch: 15 usd: 58 > Dec 27 04:37:34 *SERVER NAME* kernel: CPU 1: Hot: hi: 186, btch: > 31 usd: 156 Cold: hi: 62, btch: 15 usd: 27 > Dec 27 04:37:34 *SERVER NAME* kernel: Normal per-cpu: > Dec 27 04:37:34 *SERVER NAME* kernel: CPU 0: Hot: hi: 186, btch: > 31 usd: 151 Cold: hi: 62, btch: 15 usd: 56 > Dec 27 04:37:34 *SERVER NAME* kernel: CPU 1: Hot: hi: 186, btch: > 31 usd: 165 Cold: hi: 62, btch: 15 usd: 14 > Dec 27 04:37:34 *SERVER NAME* kernel: Free pages: 21488kB (0kB > HighMem) > Dec 27 04:37:34 *SERVER NAME* kernel: Active:832717 inactive:260329 > dirty:0 writeback:0 unstable:0 free:5372 slab:14827 mapped:650 > pagetables:14672 > 2 > Dec 27 04:37:34 *SERVER NAME* kernel: DMA free:16016kB min:3660kB > low:4572kB high:5488kB active:807232kB inactive:896400kB > present:2097152kB pages_ > scanned:3937815 all_unreclaimable? yes Dec 27 04:37:34 *SERVER NAME* > kernel: lowmem_reserve[]: 0 0 3072 3072 Dec 27 04:37:34 *SERVER NAME* > kernel: Normal free:5472kB min:5492kB low:6864kB high:8236kB > active:2523636kB inactive:144916kB present:3145728kB pag > es_scanned:7066898 all_unreclaimable? yes Dec 27 04:37:34 *SERVER > NAME* kernel: lowmem_reserve[]: 0 0 0 0 Dec 27 04:37:34 *SERVER NAME* > kernel: DMA: 1260*4kB 1258*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB > 1*512kB 0*1024kB 0*2048kB 0*4096kB = 16016kB Dec 27 04:37:34 *SERVER > NAME* kernel: Normal: 10*4kB 507*8kB 0*16kB 1*32kB 1*64kB 0*128kB > 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5472kB Dec 27 04:37:34 > *SERVER NAME* kernel: Swap cache: add 31963696, delete 31964356, find > 17660088/21209122, race 8+295 Dec 27 04:37:34 *SERVER NAME* kernel: > Free swap = 0kB Dec 27 04:37:35 *SERVER NAME* kernel: Total swap = > 2352736kB > Dec 27 04:37:35 *SERVER NAME* kernel: Free swap: 0kB > Dec 27 04:37:35 *SERVER NAME* kernel: 1310720 pages of RAM Dec 27 > 04:37:35 *SERVER NAME* kernel: 26207 reserved pages Dec 27 04:37:35 > *SERVER NAME* kernel: 66904 pages shared Dec 27 04:37:35 *SERVER > NAME* kernel: 352 pages swap cached Dec 27 04:37:35 *SERVER NAME* > kernel: Out of Memory: Kill process 26578 > (oracle) score 48132 and children. > Dec 27 04:37:35 *SERVER NAME* kernel: Out of memory: Killed process > 26578 (oracle). This says that at the time OOM was invoked, you had no free paging slots (Free swap = 0kB), no free memory (0kB HighMem), and no memory that could be reclaimed (all_unreclaimable? yes). What kicks off at that time of day? Some sort of Oracle housekeeping processes? Mark Post ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
