Re: [U2] Performance reports on 4 Terabyte files vs Distributed Files
Thanks Ross and Tony for your insights. -Baker This communication, its contents and any file attachments transmitted with it are intended solely for the addressee(s) and may contain confidential proprietary information. Access by any other party without the express written permission of the sender is STRICTLY PROHIBITED. If you have received this communication in error you may not copy, distribute or use the contents, attachments or information in any way. Please destroy it and contact the sender. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Performance reports on 4 Terabyte files vs Distributed Files
I have a customer with files at the 600 gig level, we did not go the 64 bit route (for various reasons) but made the files dynamic (this is on udt where dynamic files are held in a directory with separate data blocks as files so you do not get a file over 2 gig). Write performance is fine as it is written to all the time, however we have had instances of corruption on the file, and performing any selects takes hours, this is a fairly high specced ibm machine as well. Never played with part files, they always feel kind off old school to me for some reason. Rgds Symeon. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris Sent: 23 July 2009 02:53 To: U2 Users List Subject: Re: [U2] Performance reports on 4 Terabyte files vs Distributed Files G'day Baker, Just thinking in the sandbox, with no 4Tb databases to be found, BUT (as usual) there may be many aspects to the answer My initial thoughts would be that avoiding running a subroutine to determine part file for EVERY write and non-sequential READ would represent a saving per transaction. I would assume that such a large file is probably (should be?) well indexed, so there are likely to be some gains on that front. HOWEVER, if the file is primarily used for reporting, and the parts have been crafted with this in mind, I could see there may be situations were being able to cheat and deal directly with a part file could be advantageous. However, my gut feel (with sand grit in my teeth) is that the larger 64bit file would win out in most circumstances, but like I said, I don't have a play thing that large at hand :-) Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Baker Hughes Sent: Thursday, 23 July 2009 2:01 AM To: 'U2 Users List' Subject: [U2] Performance reports on 4 Terabyte files vs Distributed Files Does anyone have any real life examples of performance gains from converting your large file to 64 bit files? Does anyone have a performance comparison of 64 bit files vs. distributed files. I wonder if a 4 terabyte file responds better/faster than a distributed/part file? SELECTs, WRITES, other access. UV 10.2 - I guess UD has the same? Thanks, -Baker This communication, its contents and any file attachments transmitted with it are intended solely for the addressee(s) and may contain confidential proprietary information. Access by any other party without the express written permission of the sender is STRICTLY PROHIBITED. If you have received this communication in error you may not copy, distribute or use the contents, attachments or information in any way. Please destroy it and contact the sender. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Performance reports on 4 Terabyte files vs Distributed Files
From: Baker Hughes Does anyone have any real life examples of performance gains from converting your large file to 64 bit files? Does anyone have a performance comparison of 64 bit files vs. distributed files. I wonder if a 4 terabyte file responds better/faster than a distributed/part file? SELECTs, WRITES, other access. UV 10.2 - I guess UD has the same? One way to do this sort of testing yourself is to make use of testing facilities provided by hardware vendors. I know HP and IBM used to provide this service and I believe they still do. For example, when I was working at Pick Systems we would benchmark new releases of D3 at the IBM RDC in San Mateo, California. We'd run D3 with 10 thousand users, with terabytes of data, and as much memory and CPU power as was available. It was quite an experience and the result was good marketing material both for Pick Systems and for IBM. I suspect IBM would be open (for free or fee) to allowing a VAR or large end-user to load Universe or Unidata at an RDC to run a variety of tests so that you can get some real numbers on distributed files, indexes, and other configuration settings that affect performance. My recommendation would be for a VAR to do this, do a lot of different kinds of tests, and get a lot of numbers. (*grinning and waiting for the group to explode*) And then they can sell their numbers to large sites to compensate them for their efforts. Actually, this would be a good project for Spectrum. Tony Gravagno Nebula Research and Development TG@ remove.pleaseNebula-RnD.com remove.pleaseNebula-RnD.com/blog Visit PickWiki.com! Contribute! ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Performance reports on 4 Terabyte files vs Distributed Files
G'day Baker, Just thinking in the sandbox, with no 4Tb databases to be found, BUT (as usual) there may be many aspects to the answer My initial thoughts would be that avoiding running a subroutine to determine part file for EVERY write and non-sequential READ would represent a saving per transaction. I would assume that such a large file is probably (should be?) well indexed, so there are likely to be some gains on that front. HOWEVER, if the file is primarily used for reporting, and the parts have been crafted with this in mind, I could see there may be situations were being able to cheat and deal directly with a part file could be advantageous. However, my gut feel (with sand grit in my teeth) is that the larger 64bit file would win out in most circumstances, but like I said, I don't have a play thing that large at hand :-) Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Baker Hughes Sent: Thursday, 23 July 2009 2:01 AM To: 'U2 Users List' Subject: [U2] Performance reports on 4 Terabyte files vs Distributed Files Does anyone have any real life examples of performance gains from converting your large file to 64 bit files? Does anyone have a performance comparison of 64 bit files vs. distributed files. I wonder if a 4 terabyte file responds better/faster than a distributed/part file? SELECTs, WRITES, other access. UV 10.2 - I guess UD has the same? Thanks, -Baker This communication, its contents and any file attachments transmitted with it are intended solely for the addressee(s) and may contain confidential proprietary information. Access by any other party without the express written permission of the sender is STRICTLY PROHIBITED. If you have received this communication in error you may not copy, distribute or use the contents, attachments or information in any way. Please destroy it and contact the sender. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
