I don't agree. Disk access is inherently slower than RAM access. Therefore a process that makes efficient use of RAM will be faster than an equivalent algorithm making efficient use of disk.
In your case, it's just a matter of scale: 50 million records at (lets say) 14 bytes per ID plus the multivalue marker needed to build up the dynamic array. 15 * 50,000,000 = 750,000,000 bytes. That's 732,422KB, 715MB If your process is running on a modern server then this kind of op becomes practical. Assumptions: - that the dynamic array isn't using Unicode. If it is then memory reqirements double. - That you select every record - normally (presumably) it would be just a fraction? In fact isn't all of this theoretical? Using the index select / readfwd / own tests method, there's no need to build workfiles or dynamic arrays at all - simply do the tests as each record is retrieved with readfwd and then create the report / do the processing all within the same loop? Ed -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen, Rodney A (Rod) 46K Sent: 08 July 2009 12:30 To: U2 Users List Subject: Re: [U2] General guidelines on indexing When you have a file with 50 million records, it does not matter how you build the or parse the dynamic array. A well sized work file will run circles around the dynamic array. -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown Sent: Wednesday, July 08, 2009 6:12 AM To: U2 Users List Subject: Re: [U2] General guidelines on indexing > After indexing, we made a lot more use of the SETINDEX and READFWD logic in our programs. I find this curious / disappointing - is it really the case that unidata can't take the mix of indexed / unindexed dictionary items and do just as efficient a job as the code you're writing? Also, the performance of dynamic arrays need not be as much an issue as you've found. If they're built up with -1 rather than a counter then the speed penalty of adding items to a very large list is much the same as a tiny one. Then, when you loop through them for further processing use the REMOVE (or FORMLIST) command rather than a counter. The only real issue with dynamic arrays is if the machine does not have the physical memory available to hold the variable. Ed ------------------------------------------------------------------------ ------------------- Please remember to recycle wherever possible. Reduce, reuse, recycle, think do you need to print this e-mail? ------------------------------------------------------------------------ ------------------- This e-mail and any attachment(s), is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the addressee, dissemination, copying or use of this e-mail or any of its content is prohibited and may be unlawful. If you are not the intended recipient please inform the sender immediately and destroy the e-mail, any attachment(s) and any copies. All liability for viruses is excluded to the fullest extent permitted by law. It is your responsibility to scan or otherwise check this email and any attachment(s). Unless otherwise stated (i) views expressed in this message are those of the individual sender (ii) no contract may be construed by this e-mail. Emails may be monitored and you are taken to consent to this monitoring. Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 01628868 Both companies are registered in England and Wales and each has its registered office at 2 Burston Road, Putney, London, SW15 6AR. ------------------------------------------------------------------------ ------------------- _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
