Re: DFSORT memory used & paging
"R Hey" wrote in message news:... > > Why does the percent of page slots keep increasing over time. > > That got me too. First I thought we had a bug. But it appears to be a feature. > It seems that not all page-outs would be brought in, unless they are wanted. Pages are only brought in when they are wanted. Still, the page on the pagedataset remains there, so if the page has to be paged out again and it has not change, one pageout I/O is saved. They are freed when the page is deleted, since there is no reason to keep a copy anymore. So from the time of the IPL, you will see the pagedatasets filling until some balance has been reached. If you don't want this, you will probably need tons of extra memory. > > I like the idea of NOT running the big sort jobs at the same time. I do. If memory is available or can be made available, use it to eliminate I/O. Have a look at your UIC values. The pages that will be paged out, will probably be unreferenced for hours. If all this does not result in heavy page-ins, I think you are exploiting your memory efficiently. Kees. > > Rez > > Memory is the first to go & I can't remember the second one ... > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ** -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
> Why does the percent of page slots keep increasing over time. That got me too. First I thought we had a bug. But it appears to be a feature. It seems that not all page-outs would be brought in, unless they are wanted. I like the idea of NOT running the big sort jobs at the same time. Rez Memory is the first to go & I can't remember the second one ... -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
My response is not directly to this posting, just my comments to the original question. I do not have the original post. First, I have to ask what is more important to you? Is it SORTs or your online systems? I will assume online. We had problems with large sorts impacting our online. CICS is usually storage protected, so it was not impacted. We run Websphere Application Server and this was badly affected. WAS is not storage protected and uses a lot of memory. So it was a prime target for page steals. After the sort started we had high page-in rates for WAS. This was an issue. We looked at DFSORT options and decided we wanted to stop the page steals.We changed the EXPOLD parameter to zero, The default is MAX. This means that now DFSORT will only try to use available memory and not steal unreferenced pages from other work. If you are looking to make SORT fly then you may want to leave DFSORT to steal pages. But we want our on-line systems to perform and so DFSORT is a low priority. At the time of our problem we were storage constrained. Not paging, but low available frame queue. We added more memory and now have a lot free. We still do not allow DFSORT to steal pages though. On paging, I come from the school that I do not want to page. If I could restrict to very low priority work then maybe a little paging. The cost of memory means that you get more benefit from adding memory than messing around with the paging subsystem. Yes I want a paging subsystem that can handle large paging, but I do not want to use it. Its there as insurance. Also beware of free memory depletion when you take an SVCDUMP. This can hit you the same way. Terry Draper zSeries Performance Consultant w...@btopenworld.com mobile: +966 556730876 --- On Tue, 19/1/10, Staller, Allan wrote: From: Staller, Allan Subject: Re: DFSORT memory used & paging To: IBM-MAIN@bama.ua.edu Date: Tuesday, 19 January, 2010, 13:19 Page out is asynchronous and will only affect paging performance to the extent that the channel/device is not available for demand page-in. By %USE I meant page slots used, meaning increase in page out. > Why does workflow drop from %use? too much juice wasted on paging out/in. The issue is not page config, but increase in paging out/in & drop in workflow. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Page out is asynchronous and will only affect paging performance to the extent that the channel/device is not available for demand page-in. By %USE I meant page slots used, meaning increase in page out. > Why does workflow drop from %use? too much juice wasted on paging out/in. The issue is not page config, but increase in paging out/in & drop in workflow. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Reza, What does it matter if the slot count grows the way you described> the slot count does not affect performance, nor does writing those slots. Apart from keeping slot allocation efficient you should not be concerned with a slot usage less than 30%. So what you are saying is that when the large sorts run there are a lot of DB2 pages with a high UIC, and they are stolen when any jobs using a lot of memory run. That could be DFSORT, but it could also be Hiperbatch, or even LSR VSAM buffer pools. This is exactly the problem I described. So what sort of page-in rate is experienced by DB2, and what is the workload that is affected. If it is batch workload that runs after the sorts then knobbling the sorts may increase the batch critical path more than the page-in delays. I'm not actually hearing a problem. Is transaction response time unacceptable when the page-ins occur? Are the DB2 batch jobs taking 20% longer to complete? It sounds like the DB Admin has an unsavory statistic and not an actual problem. If you slow down the sorts to make his statistic go away are you going to face a new problem with additional IO load, more SORTWK Space, Sort intermediate merge and elongated Sort run times? Ron > -Original Message- > From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of > R Hey > Sent: Monday, January 18, 2010 8:49 PM > To: IBM-MAIN@bama.ua.edu > Subject: Re: [IBM-MAIN] DFSORT memory used & paging > > Ron, > > Originally DBA complained about paging, & how it reduced workflow (in our > monitor). > > I had a look to find that when the big sort jobs, some also use DB2 > after/before the sort, run at the the same time, the demand paging goes up. > > The chart was just a rough drawing to show how the demand paging looks > like, fairly flat, with a few big spikes a month. > > 60+K track is the size of some of the DS being sorted. > > These jobs run regularly, but the paging issue seems to happen when some of > the big sort ones run at the same time. > > After IPL, the slot% is 0, 2 days later is 1.5%, then a few days later may > jump > to 10% at 2230 when the sorts get to run at the same time, remain roughly > the same, then a week later jump to 16%, , then a few days later jump to > 25% (at 2345 due to sorts). > > Rez > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
"R Hey" wrote in message news:... > By %USE I meant page slots used, meaning increase in page out. > > > Why does workflow drop from %use? > > too much juice wasted on paging out/in. > > The issue is not page config, but increase in paging out/in & drop in workflow. > > So when we look at demand paging for the month, we roughly see: > > ..|...|..|... > ..|...|..|... > ..|...|..| > ||--| > > the spikes are when a few big sort jobs end up running at the same time. > > data involved is bigger than 60K tracks. > > Rgds, > Rez I don't agree with your view on this situation. You run a virtual memory OS, so paging is inherent to it. If you don't want paging, you can buy a TB of real memory or change to a real memory OS. Furthermore, if you change the DFSORT parameter such that it does not cause paging in your system, it can use much less memory and has to do much more I/O. How much juice will be spent on that I/O, which by the way is less efficient than paging I/O? Kees. ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ** -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Ron, Originally DBA complained about paging, & how it reduced workflow (in our monitor). I had a look to find that when the big sort jobs, some also use DB2 after/before the sort, run at the the same time, the demand paging goes up. The chart was just a rough drawing to show how the demand paging looks like, fairly flat, with a few big spikes a month. 60+K track is the size of some of the DS being sorted. These jobs run regularly, but the paging issue seems to happen when some of the big sort ones run at the same time. After IPL, the slot% is 0, 2 days later is 1.5%, then a few days later may jump to 10% at 2230 when the sorts get to run at the same time, remain roughly the same, then a week later jump to 16%, , then a few days later jump to 25% (at 2345 due to sorts). Rez -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Reza, You should not be concerned about Page Out rates. Page out does not interrupt the flow of a program - it's being paged out because it has not been referenced recently. If there is a page fault for that page very soon after the page out and it is still unchanged then there is no actual page-in. In terms of actual juice (MIPS) used for page-out, you would have to be generating a sustained page-out rate of several thousand pages a second to use any substantial "juice." ASM is very efficient. The usual problem with pages being pushed out by SORT is that OLTP regions that are relatively inactive during the batch run have their pages stolen, and then suffer page delays when the business opens its doors several hours later. While these pages remain unchanged on AUX they become cannon fodder for page stealing when the SORTS run the following night. In the note below you talk about page out and page in as if they produce a common delay, but they do note. You then show what appears to be some sort daily plot that shows a high DPR six times a month. Then I'm lost again because you give a Demand Paging "unit" of 60,000 tracks, and I don't know how to correlate that to anything. Do you mean that is 60Kx12x4KB blocks over a day (8.3/sec) or, an hour (200/sec), or a 15 minute RMF interval (800/sec). I think it's the reference to tracks that is throwing me. Page stealing rarely affects batch jobs themselves, as their pages usually have a low UIC while there running, and will not be used again when the job ends. Jobs that start after the sorts are unaffected because they weren't in the system when the pages were stolen. That simple logic - maybe over simple - is why I believe I have not run across DFSORT delaying batch runs. It is usually OLTP regions and long running STC that are affected by the ensuing page-ins. If you think Page-ins are causing some delay then I suggest looking at the ASID level page-in rates in the SMF type 30 interval records and see what Service Classes or Workload types are experiencing a page-in problem after the sorts are finished, and how long that rate is sustained. If your SMF interval is too long, start an RMF II Background session at 5 second intervals and look at the PIN rates for each ASID using "ASD A,A" This will give a very detailed profile of where your Page-in rate is going, and whether it is hurting one of the loved ones. Ron > -Original Message- > From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of > R Hey > Sent: Monday, January 18, 2010 4:20 PM > To: IBM-MAIN@bama.ua.edu > Subject: Re: [IBM-MAIN] DFSORT memory used & paging > > By %USE I meant page slots used, meaning increase in page out. > > > Why does workflow drop from %use? > > too much juice wasted on paging out/in. > > The issue is not page config, but increase in paging out/in & drop in > workflow. > > So when we look at demand paging for the month, we roughly see: > > ..|...|..|... > ..|...|..|... > ..|...|..| > ||--| > > the spikes are when a few big sort jobs end up running at the same time. > > data involved is bigger than 60K tracks. > > Rgds, > Rez > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
By %USE I meant page slots used, meaning increase in page out. > Why does workflow drop from %use? too much juice wasted on paging out/in. The issue is not page config, but increase in paging out/in & drop in workflow. So when we look at demand paging for the month, we roughly see: ..|...|..|... ..|...|..|... ..|...|..| ||--| the spikes are when a few big sort jobs end up running at the same time. data involved is bigger than 60K tracks. Rgds, Rez -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Is %use page dataset percent used (page slots) or page dataset percent
busy (paging IO)?
Page slot percent used should have no impact on performance unless there
is significant demand paging (page in from DASD) occurring. Check the
RMF post processor reports.
25% page slot utilization is not a big issue. The last major update to
the paging subsystem algorithms ("block paging" circa 1990?) the
recommended target utilization was 30-35%.
Even exceeding the 30% will not be a disaster, only less efficient
paging. If your demand page rate (page in from DASD) is still low, this
is not a problem.
The only issue left is total page slot utilization. This can be easily
handled by keeping a spare pageds around to PAGEADD if necessary.
HTH,
When we get a few big sort jobs running, our page DS %use increases.
Which cause workflow to drop.
Page %USE doesn't drop much, but it can go up if we get another run of a
few big sort jobs later.
After a month, we could end up with %USE of 25 (with 3-4 big concurrent
SORT runs).
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
"R Hey" wrote in message news:... > Thanks guys. > > When we get a few big sort jobs running, our page DS %use increases. > Which cause workflow to drop. Why does workflow drop from %use? > > Page %USE doesn't drop much, but it can go up if we get another run of a > few big sort jobs later. > > After a month, we could end up with %USE of 25 (with 3-4 big concurrent > SORT runs). > Like you should design the performance of your paging configuration on worst case situations, that you might only encounter once a year, you should also design the size of your pagedatasets on worst case situations. > Rgds, > Rez > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ** -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Thanks guys. When we get a few big sort jobs running, our page DS %use increases. Which cause workflow to drop. Page %USE doesn't drop much, but it can go up if we get another run of a few big sort jobs later. After a month, we could end up with %USE of 25 (with 3-4 big concurrent SORT runs). Rgds, Rez -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Good points Kees. The original question didn't indicate a paging "problem", just an increase in page DS use and demand paging. So yes, tuning the paging subsystem is certainly a valid approach. I was speaking mainly from a DFSORT perspective on ways to minimize paging. Usually a small amount of demand paging isn't a problem, but then again it depends on what workload is doing it. IBM Mainframe Discussion List wrote on 01/15/2010 03:47:26 AM: > > Dave, > > I would have expected you to also mention a different approach: check > the paging configuration. > > During the years I have become quite confident about the intelligence > DFSORT has developped to use the available system resources, without > causing problems in the system. One of the presumptions is the > installation has a adequate paging configuration. We had a similar > situation lately where we had paging performance problemns during DFSORT > jobs. We have ample storage and the large DFSORT jobs decided to take a > couple of GB central storage for its sort. This caused a large page out > activity (like you described above) with I/O problems. > > After investigating the situation, the conclusion could only be that > either DFSORT miscalculated the amount of available storage *or* the > paging subsystem was not able to handle a substantial page-out load. We > looked at our paging configuration and the spreading of the volumes over > the internal ESS ranks and concluded that it contained several hotspots. > We reconfigured the paging volumes and never saw the problem again. > > Everybody has read many years ago the recommandation for his/her paging > configuration to spread it over as many volumes, channels and control > units as possible, said "yes, of course", and continued with other > things. A under-optimal paging configuration works well most of the time > except in special situations like large dumps and yes, large DFSORTs. > > I think that also in Rez's case, both sides of the problem should be > examined and if the paging configuration is insufficient *and* he is not > able to correct this, the other option is to limit DFSORT's use of > storage. > > Kees. > ** > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee > only. If you are not the addressee, you are notified that no part > of the e-mail or any attachment may be disclosed, copied or > distributed, and that any other action related to this e-mail or > attachment is strictly prohibited, and may be unlawful. If you have > received this e-mail by error, please notify the sender immediately > by return e-mail, and delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries > and/or its employees shall not be liable for the incorrect or > incomplete transmission of this e-mail or any attachments, nor > responsible for any delay in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal > Dutch Airlines) is registered in Amstelveen, The Netherlands, with > registered number 33014286 > ** > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
Excellent observation, Kees! Has anyone given thought to some additional health checks along these lines similar to the couple datasets SPOF checks? Also, happy values for DFSORT? David, any opportunities here? :-) Bob -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Vernooij, CP - SPLXM Sent: Friday, January 15, 2010 3:47 AM To: IBM-MAIN@bama.ua.edu Subject: Re: DFSORT memory used & paging "David Betten" wrote in message > snipped Dave, I would have expected you to also mention a different approach: check the paging configuration. During the years I have become quite confident about the intelligence DFSORT has developped to use the available system resources, without causing problems in the system. One of the presumptions is the installation has a adequate paging configuration. We had a similar situation lately where we had paging performance problemns during DFSORT jobs. We have ample storage and the large DFSORT jobs decided to take a couple of GB central storage for its sort. This caused a large page out activity (like you described above) with I/O problems. After investigating the situation, the conclusion could only be that either DFSORT miscalculated the amount of available storage *or* the paging subsystem was not able to handle a substantial page-out load. We looked at our paging configuration and the spreading of the volumes over the internal ESS ranks and concluded that it contained several hotspots. We reconfigured the paging volumes and never saw the problem again. Everybody has read many years ago the recommandation for his/her paging configuration to spread it over as many volumes, channels and control units as possible, said "yes, of course", and continued with other things. A under-optimal paging configuration works well most of the time except in special situations like large dumps and yes, large DFSORTs. I think that also in Rez's case, both sides of the problem should be examined and if the paging configuration is insufficient *and* he is not able to correct this, the other option is to limit DFSORT's use of storage. Kees. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
"David Betten" wrote in message news:... > > > > DFSORT memory used & paging > > > > R Hey > > > > to: > > > > IBM-MAIN > > > > 01/14/2010 09:20 PM > > > > Sent by: > > > > IBM Mainframe Discussion List > > > > Please respond to IBM Mainframe Discussion List. > > > > Hi Frank/List, > > > > My client has some very big SORT jobs. > > When a few run at the the same time, the page DS USE% increases > > (demanding paging). > > This is likely due to DFSORT creating large memory objects, Hiperspaces > or Dataspaces for sorting in memory and that's causing pages to be > stolen from other workloads which then get paged out. > > > > > Q1- > > How do I make sure BIG sort jobs don't use too many frames, > > when they are hard to get? > > DFSORT has installation defaults that can limit the storage used > by all concurrent sorts. EXPOLD controls the amount of "old" (not > recently referenced) frames that are considered in DFSORT's calculations > of storage available for sorting in memory. The shipped > default is MAX. You can change this if you want to limit that number of > old frames DFSORT uses which in turn would reduce the page stealing and > demand paging. > For example, if you set EXPOLD=0, then DFSORT would only consider available > frames > in its calculations of storage availabe for memory objects, Hiperspaces and > Dataspaces. > > > > > Q2- > > Given the sort options we use, is it true that > > SORT will use TMAXLIM size of storage, > > then if data is too big, it will increase the storage use to max of DSA > > size? > > So it never goes over DSA size? > > This is correct. > > > > > Q3- > > If I get > > ICE199I 0 MEMORY OBJECT STORAGE USED = 3463M BYTES > > it means 3.4GB of data was sorted in memory, right? > > > > Correct. > > > Q4- > > If YES to both Q2/Q3, then how is 3GB of data is sorted in 64M of memory > > (our DSA) > > If, not, how does it work? > > DFSORT will use up to 64MB of virtual storage for buffers, control blocks, > etc. It can also create Hiperspaces or memory objects to be used as > intermediate storage for sorting the data. The intermediate storage is > limited by DFSORT installation defaults such as EXPMAX, EXPOLD, HIPRMAX, > MOSIZE, DSPSIZE, etc. It's also limited by the amount of available storage > at the time the sort executes. > > > > > > Q5- > > We use SIZE=MAX, but the job has SIZE=32259547 (it can be different for > > another run). > > Is this because the size of data being sorted determines the actual SIZE > > used? > > Yes, DFSORT's Dynamic Storage Adjustment (DSA) alogorithm calculates the > optimal amount of main storage for a sort. The DSA installation default is > a limit. So DFSORT will use minimum of the calculated value or the DSA > limit. > > > Dave, I would have expected you to also mention a different approach: check the paging configuration. During the years I have become quite confident about the intelligence DFSORT has developped to use the available system resources, without causing problems in the system. One of the presumptions is the installation has a adequate paging configuration. We had a similar situation lately where we had paging performance problemns during DFSORT jobs. We have ample storage and the large DFSORT jobs decided to take a couple of GB central storage for its sort. This caused a large page out activity (like you described above) with I/O problems. After investigating the situation, the conclusion could only be that either DFSORT miscalculated the amount of available storage *or* the paging subsystem was not able to handle a substantial page-out load. We looked at our paging configuration and the spreading of the volumes over the internal ESS ranks and concluded that it contained several hotspots. We reconfigured the paging volumes and never saw the problem again. Everybody has read many years ago the recommandation for his/her paging configuration to spread it over as many volumes, channels and control units as possible, said "yes, of course", and continued with other things. A under-optimal paging configuration works well most of the time except in special situations like large dumps and yes, large DFSORTs. I think that also in Rez's case, both sides of the problem should be examined and if the paging configuration is insufficient *and* he is not able to correct this, the other option is to limit DFSORT's use of storage. Kees. ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by r
Re: DFSORT memory used & paging
> > Recommend you start at the IBM DFSORT home-page where there are several > documentation and supplemental technical references on DFSORT and > memory usage. > > Scott Barry > SBBWorks, Inc. > > Suggested Google advanced search argument on this topic: > > dfsort memory usage site:ibm.com > Good point Scott. I should've included that in my original post. There's also the DFSORT Tuning Guide which is very helpful. Dave Betten DFSORT Development, Performance Lead IBM Corporation email: bet...@us.ibm.com DFSORT/MVSontheweb at http://www.ibm.com/storage/dfsort/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
On Thu, 14 Jan 2010 20:19:59 -0600, R Hey wrote: >Hi Frank/List, > >My client has some very big SORT jobs. >When a few run at the the same time, the page DS USE% increases >(demanding paging). > >Q1- >How do I make sure BIG sort jobs don't use too many frames, >when they are hard to get? > >Q2- >Given the sort options we use, is it true that >SORT will use TMAXLIM size of storage, >then if data is too big, it will increase the storage use to max of DSA >size? >So it never goes over DSA size? > >Q3- >If I get >ICE199I 0 MEMORY OBJECT STORAGE USED = 3463M BYTES >it means 3.4GB of data was sorted in memory, right? > >Q4- >If YES to both Q2/Q3, then how is 3GB of data is sorted in 64M of memory >(our DSA) >If, not, how does it work? > >Q5- >We use SIZE=MAX, but the job has SIZE=32259547 (it can be different for >another run). >Is this because the size of data being sorted determines the actual SIZE >used? > > >TIA, >Rez > > >Our sort option are: > >MODULE ICEAM1 > - - - - - - - - - - - >ARESALL0 >ARESINVNOT APPLICABLE > - - - - - - - - - - - >DSA64 > - - - - - - - - - - - >DYNALOC(SYSALLDA,3) > * (SYSDA,4) > - - - - - - - - - - - >EXPMAX MAX >EXPOLD MAX >EXPRES 0 > - - - - - - - - - - - >HIPRMAXOPTIMAL > - - - - - - - - - - - >MAXLIM 1048576 >MINLIM 450560 >MOSIZE MAX > - - - - - - - - - - - >OVERRGN65536 >RESALL 4096 >RESET YES >RESINV NOT APPLICABLE > - - - - - - - - - - >SIZE MAX > - - - - - - - - - - >TMAXLIM6291456 > - - - - - - - - - - >VIONO > > >joblog had: > >ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION >MODULE >SELECTED >ICE088I 5 PAK328 .SORT305 . , INPUT LRECL = 650, BLKSIZE = 27950, TYPE = >FB >ICE093I 0 MAIN STORAGE = (MAX,32259547,32259547) >ICE156I 0 MAIN STORAGE ABOVE 16MB = (32122707,32122707) >ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 >,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256 > >ICE128I 0 OPTIONS: >SIZE=32259547,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=AB >END,MS >GDDN=SYSOUT >ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO >,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC= >(SCRATCH ,004),ABCODE=MSG >ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 >,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,STIMER=Y,COBEXIT=COB2 > >ICE131I 0 OPTIONS: >TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DS >A=64 >ICE132I 0 OPTIONS: >VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE >,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N >ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX >,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX >ICE235I 0 OPTIONS: NULLOUT=RC0 >ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT >ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN >ICE750I 0 DC 3576062750 TC 0 CS DSVVV KSZ 36 VSZ 36 >ICE752I 0 FSZ=5501635 RC IGN=0 E AVG=652 0 WSP=4658983 C DYN=0 >0 >ICE751I 1 BA-K22788 BB-K24705 BC-K24705 E8-K90013 >ICE090I 0 OUTPUT LRECL = 650, BLKSIZE = 27950, TYPE = FB (SDB) >ICE080I 0 IN MAIN STORAGE SORT >ICE055I 0 INSERT 0, DELETE 0 >ICE054I 0 RECORDS - IN: 5501623, OUT: 5501623 >ICE134I 0 NUMBER OF BYTES SORTED: 3576054950 >ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 0 , TRACKS USED: >0 >ICE199I 0 MEMORY OBJECT STORAGE USED = 3463M BYTES >ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES >ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES >ICE052I 0 END OF DFSORT > Recommend you start at the IBM DFSORT home-page where there are several documentation and supplemental technical references on DFSORT and memory usage. Scott Barry SBBWorks, Inc. Suggested Google advanced search argument on this topic: dfsort memory usage site:ibm.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DFSORT memory used & paging
> > DFSORT memory used & paging > > R Hey > > to: > > IBM-MAIN > > 01/14/2010 09:20 PM > > Sent by: > > IBM Mainframe Discussion List > > Please respond to IBM Mainframe Discussion List. > > Hi Frank/List, > > My client has some very big SORT jobs. > When a few run at the the same time, the page DS USE% increases > (demanding paging). This is likely due to DFSORT creating large memory objects, Hiperspaces or Dataspaces for sorting in memory and that's causing pages to be stolen from other workloads which then get paged out. > > Q1- > How do I make sure BIG sort jobs don't use too many frames, > when they are hard to get? DFSORT has installation defaults that can limit the storage used by all concurrent sorts. EXPOLD controls the amount of "old" (not recently referenced) frames that are considered in DFSORT's calculations of storage available for sorting in memory. The shipped default is MAX. You can change this if you want to limit that number of old frames DFSORT uses which in turn would reduce the page stealing and demand paging. For example, if you set EXPOLD=0, then DFSORT would only consider available frames in its calculations of storage availabe for memory objects, Hiperspaces and Dataspaces. > > Q2- > Given the sort options we use, is it true that > SORT will use TMAXLIM size of storage, > then if data is too big, it will increase the storage use to max of DSA > size? > So it never goes over DSA size? This is correct. > > Q3- > If I get > ICE199I 0 MEMORY OBJECT STORAGE USED = 3463M BYTES > it means 3.4GB of data was sorted in memory, right? > Correct. > Q4- > If YES to both Q2/Q3, then how is 3GB of data is sorted in 64M of memory > (our DSA) > If, not, how does it work? DFSORT will use up to 64MB of virtual storage for buffers, control blocks, etc. It can also create Hiperspaces or memory objects to be used as intermediate storage for sorting the data. The intermediate storage is limited by DFSORT installation defaults such as EXPMAX, EXPOLD, HIPRMAX, MOSIZE, DSPSIZE, etc. It's also limited by the amount of available storage at the time the sort executes. > > Q5- > We use SIZE=MAX, but the job has SIZE=32259547 (it can be different for > another run). > Is this because the size of data being sorted determines the actual SIZE > used? Yes, DFSORT's Dynamic Storage Adjustment (DSA) alogorithm calculates the optimal amount of main storage for a sort. The DSA installation default is a limit. So DFSORT will use minimum of the calculated value or the DSA limit. > > > TIA, > Rez > > > Our sort option are: > > MODULE ICEAM1 > - - - - - - - - - - - > ARESALL0 > ARESINVNOT APPLICABLE > - - - - - - - - - - - > DSA64 > - - - - - - - - - - - > DYNALOC(SYSALLDA,3) >* (SYSDA,4) > - - - - - - - - - - - > EXPMAX MAX > EXPOLD MAX > EXPRES 0 > - - - - - - - - - - - > HIPRMAXOPTIMAL > - - - - - - - - - - - > MAXLIM 1048576 > MINLIM 450560 > MOSIZE MAX > - - - - - - - - - - - > OVERRGN65536 > RESALL 4096 > RESET YES > RESINV NOT APPLICABLE > - - - - - - - - - - > SIZE MAX > - - - - - - - - - - > TMAXLIM6291456 > - - - - - - - - - - > VIONO > > > joblog had: > > ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION > MODULE > SELECTED > ICE088I 5 PAK328 .SORT305 . , INPUT LRECL = 650, BLKSIZE = 27950, TYPE = > FB > ICE093I 0 MAIN STORAGE = (MAX,32259547,32259547) > ICE156I 0 MAIN STORAGE ABOVE 16MB = (32122707,32122707) > ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 > ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256 > > ICE128I 0 OPTIONS: > SIZE=32259547,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=AB > END,MS > GDDN=SYSOUT > ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO > ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC= > (SCRATCH ,004),ABCODE=MSG > ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 > ,CHECK=N,WRKREL=Y,OUTREL=Y,CKPT=N,STIMER=Y,COBEXIT=COB2 > > ICE131I 0 OPTIONS: > TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DS > A=64 > ICE132I 0 OPTIONS: > VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE > ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N > ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX > ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX > ICE235I 0 OPTIONS: NULLOUT=RC0 > ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT > ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN > ICE750I 0 DC 3576062750 TC 0 CS DSVVV KSZ 36 VSZ 36 > ICE752I 0 FSZ=5501635 RC IGN=0 E AVG=652 0 WSP=4658983 C DYN=0 > 0 > ICE751I 1 BA-K22788 BB-K24705 BC-K24705 E8-K90013 > ICE090I 0 OUTPUT LRECL = 650, BLKSIZE = 27950, TYPE = FB (SDB) > ICE080I 0 IN MAIN STORAGE SORT > ICE055I 0 INSERT 0, DELETE 0 > ICE054I 0 RECORDS - IN: 5501623, OUT: 5501623 > ICE134I 0 NUMBER OF BYTES SORTED: 3576054950 > ICE165I 0
