On Thu, Nov 7, 2019 at 6:16 PM van Sleeuwen, Berry <[email protected]> wrote: > > Hi Christian, > > Indeed I did expect to easily add disks to the machine. Our SUSE machines > have a lot of DASD and run in pretty small machines. So I was very surprised > to see this problem. > > As for process, in top [kswapd0] showed the large CPU load. Also the memory > in top showed all memory was full and the eventhough it's just an 'empty' > machine there is swap space usage increased. I didn't list iotop, if anything > I was already happy to be able to start top. But since [kswapd0] was the > active process I assumed swapping is the cause of the high IO load. In z/VM I > could see over 4000 IO/s on DASD (performance toolkit panel FCX115).
Ok, lets for now assume that all CPU and I/O was driven by swapping. > Meanwhile I have moved the installed image from the installer (virtual > storage 2GB) into the production configuration (virtual storage 32GB). Now I > can activate disks without any problem, but it also shows why it was a > problem in the installer system. > > After boot: (29 ECKD disks) > Tasks: 136 total, 1 running, 60 sleeping, 0 stopped, 0 zombie > KiB Mem : 32807736 total, 31001912 free, 1637604 used, 168220 buff/cache > KiB Swap: 719852 total, 719852 free, 0 used. 30877584 avail Mem Unfortunately these stats means nothing as it has all sorts of caching and other things that would be pushed out of memory before swapping or even an OOM. > > So the with the primary disk configuration the machine already assigned > 1.6GB, given the available memory in the installer system that was already > too much. After I activated another 23 disks, the used memory is now 2.4GB > and after activating another 7 disk it's 2.6GB. > > I do have the OOM list in /var/log/syslog but I am unsure what to look for. Usually an OOM includes something like a top-scorer list of apps and their consumption. It is a table starting with a header like this (details depend on the kernel version) [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name But actually the whole section starting with "Out of memory: Kill ..." might help. @Frank - have you recently tried many devices on a small system (or could you give it a try) and saw something similar? > Met vriendelijke groet/With kind regards/Mit freundlichen Grüßen, > Berry van Sleeuwen > Flight Forum 3000 5657 EW Eindhoven > > -----Original Message----- > From: Linux on 390 Port <[email protected]> On Behalf Of Christian > Ehrhardt > Sent: Thursday, November 07, 2019 4:07 PM > To: [email protected] > Subject: Re: Ubuntu 18-04 unable to add DASD > > On Thu, Nov 7, 2019 at 3:57 PM van Sleeuwen, Berry > <[email protected]> wrote: > > > > Hi All, > > > > I have installed an Ubuntu 19-04 but as it turns out the application > > requires version 18-04. So I have installed a new 18-04 system. > > > > Next I tried to move the LVM from the 19-04 machine into the 18-04 machine. > > But I can't get the disks online. The first few disks are fine but at a > > certain point the server crashes. During "chccwdev -e" at some point all > > memory is exhausted and I see a lot of CPU and IO. Eventually the > > OOM-killer kicks in to kill various processes. The server is no longer > > responding and only a reboot (force and xautolog) will get the server back > > online. > > Hi Berry, > I have myself enabled hundreds of dasds without such issues, so it must be > something special to your system/config that we have to find out. > We all know that chccwdev shouldn't do any I/O other than to the entries in > /sys so I'm wondering what goes on. > > You said you have a lot of CPU+IO and then the OOM-killer kicks in. > Can you outline > a) which processes spin when being CPU intensive - gather with e.g. > top in batch mode > b) which processes drive that I/O - gather with iotop > (btw - do you see that I/O in VM or in Linux) > c) when the OOM killer kicks in (or if you can before) which processes > consume huge chunks of memory? > If you have nothing else, at least the OOM report that dmesg holds should > have a list. > > From there we can start to gather ideas what it might be. > > Kind Regards, > Christian > > This e-mail and the documents attached are confidential and intended solely > for the addressee; it may also be privileged. If you receive this e-mail in > error, please notify the sender immediately and destroy it. As its integrity > cannot be secured on the Internet, Atos’ liability cannot be triggered for > the message content. Although the sender endeavours to maintain a computer > virus-free network, the sender does not warrant that this transmission is > virus-free and will not be liable for any damages resulting from any virus > transmitted. On all offers and agreements under which Atos Nederland B.V. > supplies goods and/or services of whatever nature, the Terms of Delivery from > Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be > promptly submitted to you on your request. > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or visit > http://www2.marist.edu/htbin/wlvindex?LINUX-390 -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www2.marist.edu/htbin/wlvindex?LINUX-390
