Hi Kevin, Thanks for the input. It turns out that qstat is slow anyways even w/o the xml output, and it takes up a bit of memory in the process also.
120,000 jobs in the system, qstat with xml output consumes 1.5 GB ram resident and virtual as seen in top. Clearly, qstat is less than optimal in managing memory when xml output is desired. Without xml output, qstat consumes approx 700 MB of ram, which is ok....I guess... The speed to process qstat output w/o xml is still slow, and with xml is approx 20% slower ( for 120,000 jobs in the system...) So, there may be no real way to speed the querying process up w/o an investment in time in looking at the qstat code and how it handles processing. It may be tied to the scheduler so we may be kinda out of luck here. One question is: Will going to Univa Grid Engine and using the new database spooling help speed things up? One way to find out. I'll look into obtaining the new code for testing to see if indeed it is faster in handling qstat requests when 120,000 jobs are in the system. Thanks, Ed -----Original Message----- From: Kevin Buckley [mailto:[email protected]] Sent: Wednesday, July 31, 2013 02:31 AM To: 'Ed Lauzier' Cc: [email protected], [email protected] Subject: Re: [gridengine users] qstat xml output performance On 31 July 2013 15:42, Ed Lauzier wrote: > With 120,000 jobs in the system, it takes 33 seconds to process the > following output: > > qstat -u "*" -F > "num_proc,mem_free,mem_used,virtual_free,cpu,load_ave,load_short,load_medium,load_long" > -r -ext -pri -xml > myqstatxmlfile.xml > ... > Are there any suggestions to reduce the time to process the qstat xml > output? > Do things speed up if you don't try to force the qstat command to limitthe > output fields and just do a plainqstat -u "*" -F-r -ext -pri -xml > > myqstatxmlfile.xml > to get everything and then process the XML (with XSLT or grep) to get what > youactually want? > Kevin M. Buckley > eScience ConsultantSchool of Engineering and Computer Science > Victoria University of Wellington > New Zealand
_______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
