Thank you for all the help with this Colin. Your findings appear to have obvious benefits to both; organizations concerned about high z/OSMF CPU associated with software maintenance (due to a shortage of specialty engine resources) and those interested in ZOWE but turned off by the grossly inefficient thread pool management when z/OSMF is part of the software stack.
We will test your thread pool recommendations/circumventions this week, and hopefully IBM will make this conglomeration resemble a well tuned system in the near future. Thanks again, Mike -----Original Message----- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Colin Paice Sent: Sunday, July 18, 2021 2:30 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Serverpac installs January 2022 and beyond (+hungry ZOWE concerns) Caution! This message was sent from outside your organization. I've been working with the performance person... I'll raise an RFE (or two) and post the number here. I also see lots of "set_timer" ( about 100 a second!) and lots of pause-resumes. I havent pinned it down yet. Maybe each thread wakes up to see if it has work to do (rather than a post-wait model)! Im guessing it comes from pthread_cond_timedwait64(..) in java threads, but cant get closer than that. There is also a lot of "stat" activity looking for info on a file perhaps, but havent tracked that down either. I think IBM could reduce the CPU significantly if they had a cross lab team to look into it. Most of the activities I see in the trace should not be there for a well tuned system! Colin On Sun, 18 Jul 2021 at 17:13, Mike Hochee <mike.hoc...@aspg.com> wrote: > Wow, really nice work Colin! Addresses root cause for the high z/OSMF > CPU, and is very neatly summed... > https://colinpaice.blog/2021/06/27/i-cut-the-cpu-cost-of-doing-nothing > Is there an associated RFE? > > Thank much, > Mike > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] > On Behalf Of Colin Paice > Sent: Sunday, July 18, 2021 4:32 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: Serverpac installs January 2022 and beyond (+hungry ZOWE > concerns) > > Caution! This message was sent from outside your organization. > > If z/OSMF uses a lot of CPU when idle. You should increase the number > of threads. See here I cut the CPU cost of doing nothing < > https://colinpaice.blog/2021/06/27/i-cut-the-cpu-cost-of-doing-nothing/>. > The default pool size is 100 - I found I needed 300 threads to avoid > all the expanding and shrinking of the thread pool (create threads, > lots of getmains... lots of freemains, delete threads: repeat). Ive > suggested that z/OSMF development document/fix this. > Colin > > On Sat, 17 Jul 2021 at 21:29, Mike Hochee <mike.hoc...@aspg.com> wrote: > > > Thank you Ed, excellent suggestion. > > > > I too have felt the z/OSMF cpu cycles were exorbitant, but assuming > > you have sufficient zIIP capacity and ZZ=YES, which now appears to > > be the default, then high cycles might in reality be more of a > > perception problem if a person is looking at a generalized cpu > > bucket (as I was) rather than GP, zIIP, and zAAP contributions individually. > > > > Another item, in the context of z/OSMF resource utilization as part > > of the ZOWE software stack is what I regarded as high EXCP counts > > when the system is idling. Over a 30hr period this translated to > > approximately 300 EXCPs/second from z/OSMF. I have heard from two > > sources that this may be due to a rather primitive technique used by > > ZOWE to check for new work requests - that of interrogating data > > sets for additions/changes. I would think some flavor of wait-post > > would be far more efficient. Has anyone else noticed this behavior > > or better yet, > aware of a fix for it? > > > > Added Colin Paice's post to the end of this thread since it got > > dropped and seemed quite relevant. > > > > Thanks much, > > Mike > > -----Original Message----- > > From: IBM Mainframe Discussion List > > [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Ed Jaffe > > Sent: Saturday, July 17, 2021 7:24 AM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: Re: Serverpac installs January 2022 and beyond > > > > Caution! This message was sent from outside your organization. > > > > On 7/17/2021 3:46 AM, Brian Westerman wrote: > > > Asking a site that is able to function within their requirements > > > and > > existing SLA's to upgrade their box to more than 4x the existing > > size just to run z/OSMF is just never going to be economically feasible. > > Even on sites with larger machines, they may not have the extra > > capacity to provide z/OSMF with what it needs to function properly. > > > > Since z/OSMF is a Java application, there is no need to upgrade the > > box at all in the classic sense of increasing its MSU capacity. > > > > What you do instead is purchase/enable a single zIIP engine, share > > it among all z/OS LPARs via the HMC, and set ZZ=YES (zAAP on zIIP) > > in IEASYSxx on z/OS. > > > > Voila! z/OSMF runs like a champ and your software bill does not go > > up one iota. In fact, it could go down slightly due to zIIP > > exploitation by other CPU-hungry products. > > -- > > Phoenix Software International > > Edward E. Jaffe > > 831 Parkview Drive North > > El Segundo, CA 90245 > > https://www.phoenixsoftware.com/ > > > > > > From: IBM Mainframe Discussion List > > [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Colin Paice > > Sent: Saturday, July 17, 2021 8:43 AM > > To: IBM-MAIN@LISTSERV.UA.EDU > > Subject: Re: Serverpac installs January 2022 and beyond > > > > One issue is that z/OSMF is expensive in CPU. I noticed in the > > system trace, that there are 500 storage requests (eg > > getmain/freemain) for each http rest message coming in. Getting rid > > of these expensive requests would reduce the costs. I think all of > > these come from below the Java level, such as BPX* modules. > > For example, write an SMF record from C requires a getmain .. write > > SMF record - free main. > > > > Ive blogged > > < > > https://colinpaice.blog/2020/12/21/a-practical-guide-to-getting-z-os > > mf > > -working/ > > > > > on getting z/OSMF working including digital certificates. > > Unfortunately > I > > dont think they have designed it properly. Other products have the > > "keystore" (keyring), containing just the private key for the > > server, and the "trust store" (keyring) containing the public > > certificate needed to validate any certificates sent to the server. > > This trust store can be used by all products (Sysplex wide). z/OSMF > > just uses one, combined, store(keying). This means I cant just > > reuse my existing certificate set up, and so I have to have a > > dedicated keyring for each z/OSMF. This makes the setup much > > harder, and makes administration hard. It is not much code to fix > > it ( I implemented it > in my Java server). > > > > I think that z/OSMF could be great for the younger generations to > > get up to speed. But it needs to be improved to make it low cost > > and easy to install and configure (as in get it into production, > > rather than just get it started). > > > > Colin > > > > -------------------------------------------------------------------- > > -- For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO > > IBM-MAIN > > > > -------------------------------------------------------------------- > > -- For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO > > IBM-MAIN > > > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, send > email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, send > email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
