Most likely your problem lies somewhere hidden within the code.
I encountered a similar problem at a government site, where I was as a
contractor confronted with a batch process, that took 18 hours! to run.
My task was to develop a new program using the same logic for a
calculation. I didn't want to re-invent the wheel, so I copied the code
into my program. It also took 18 hours to run, so I went through the
code of that batch process systematically, and found the "fault" after
less than a day in a small subroutine in a loop about 10 levels deep,
where one of the largest files in the system was selected "with @id
like..." .
I suggested to index the file, but the project leader of the department
meant, I was wasting his and company time, they had another contractor
in before spending 2 month on the issue, who couldn't find anything
wrong with the code or the logic, this file would be used all over the
system, file and subroutine were the property of another department, and
the primary index would take care of that anyway.
I knew he was wrong and tried it nevertheless on the test system, and
the process finished in less than 20 minutes!
Needless to say, that my contract got cancelled a short time later.
Mecki
rbl000 schrieb:
We are a UniVerse shop using rel.10.1.15 on HPUX. We have a particular file
that is quite large and central to many if not most activities. Unix file size
limitations on older hardware and the OS dictated that this file be created as
a 10 part multipart file. Recently we converted it to a single part file with
expectations of decreased access times. Our assumption was there would now be
a single hash to access the data instead of the multipart algorithm to
determine the part and then the hash into the physical part file. After the
changeover many of our batch processes against this file were much slower
rather than being slightly faster.
The file is on an EMC array which is striped over 32 logical disks. The sizing of the file does not seem to be poor.
We isolated a copy of the file on a test system and ran 2 batch processes against the single part file as a baseline. We then restored the file and converted it back to a 10 part file with the original parting algorithms we wrote and re-ran the same two batch processes. The results were:
Single Part Multi-part
Run time Run time
Batch Process A 00:59:02 00:35:26
Batch Process B 05:42:33 00:28:00
o All ten parts are the same size which is about 1/10 of the single part. FILE-STATs of the 10 parts seem to indicate hashing distribution to be about the same as with the single part file, i.e. the number of groups > 100% full on the multipart is roughly 1/10 of those in the single part file and the same for each of the other %full categories.
o Both processes are multi-threaded, i.e. they each have a controlling "parent"
program that phantoms 36 "child" processes to do the real work.
o Process A's parent selects the file and builds a select list of the
entire file. It then builds child lists, one for each child, by dividing the
large list up into equal sized chunks. These child lists are created in
file/group order. After creating the lists it phantoms its children.
o Process B re-uses the lists built by process A, and immediately
phantoms its children.
o Process A's children read every record in the list, apply filtering
criteria, and then either update and write the record or release it. (Only a
small percentage meets the criteria to be updated.)
o Process B's children read every record in the list, update certain
attributes and then write every record.
o The test machine has 8 processors. The production machine where we
first observed this has 16 processors.
We tried another very simple batch process that was single threaded and that
read, updated, and wrote every record. We created a single part and multi-part
version of the file as we had in the above tests. We ran the batch process
against each of them and the results were:
Single Part Multi-part
Run time Run time
00:26:45 00:29:40
Needless to say while the single threaded access conformed to our original
expectations, it did little to enlighten us other than to point towards a link
between multi-threading and multi-parting.
Can anyone shed any light on what we are missing that explains why the multi-part file is so much faster?
Any insights would be welcome and appreciated.
Richard Lewis
Nu Skin Enterprises
_______________________________________________
No banners. No pop-ups. No kidding.
Make My Way your home on the Web - http://www.myway.com
-------
u2-users mailing list
[email protected]
To unsubscribe please visit http://listserver.u2ug.org/
-------
u2-users mailing list
[email protected]
To unsubscribe please visit http://listserver.u2ug.org/
