RE: 2 gig limits
Speaking of 2GB limits, is there a way to get FAST to change it from one to the other? I'd swear I had FAST (v. 5) handle this once, but now I can't find it. UV's RESIZE has 32BIT 64BIT keywords that FAST could theoretically use. - cds -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
On the version of Unix we're running (SCO 3.2), you can't even create a FILESYSTEM over 2Gb - we had to partition our 4Gb drives in two ... (that dates the system :-) Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Logan, David (SST - Adelaide) Sent: 10 February 2004 20:52 To: U2 Users Discussion List Subject: RE: 2 gig limits It would have to be a 64bit file, there are no exceptions as this is a limitation brought on by the size of a number. (I think a couple of earlier posters had the numbers involved) therefore you literally cannot create a file larger than 2Gb with 32 bit addressing. *** This transmission is intended for the named recipient only. It may contain private and confidential information. If this has come to you in error you must not act on anything disclosed in it, nor must you copy it, modify it, disseminate it in any way, or show it to anyone. Please e-mail the sender to inform us of the transmission error or telephone ECA International immediately and delete the e-mail from your information system. Telephone numbers for ECA International offices are: Sydney +61 (0)2 9911 7799, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 and New York +1 212 582 2333. *** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
2 gig limits
Gentlemen, I was having a walk-around this Unidata system. I noticed that some of the files are approaching 2 gb and a couple of files are over 2 gb. Is there a future problem looming. What is the 2 gig limits mentioned in some of the email's? Dave Raven Mobile(949) 228 2224 e Fax (815)4259364 P.O. Box 17811, Irvine CA 92623-7811 -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: 2 gig limits
I did not originally notice the OP stated they were a Unidata platform. My distributed file comments were related to UniVerse. However, when designing a UniVerse distributed file, it is wise to pick the number of part files that will keep each part file size *well below* the 2GB limit. Based on how uniform of a distribution your part file algorithm yields, and the total amount of data that needs to be stored, I'd set the goal of each part file to be about 1GB (or less) in size. By doing this, you'd have another gigabyte (or more) of available space in each part file to accommodate growth. If in doubt, I'd recommend using a larger number of part files rather than fewer of them for a given distributed file. For administration ease, I like to use distributed files where I can resize (static/hashed files) each part file individually on an as-need basis. It's much easier to find weekend time to resize one or more 1GB files (or smaller), than to find a single big enough weekend time slot to resize a single 64bit file that contains 30GB of data. Additionally, the various individual part files can be spread over multiple disk spindles, so disk I/O can be optimized. HTH, ken === In a message dated 2/10/2004 12:24:18 PM Eastern Standard Time, [EMAIL PROTECTED] writes: Subj: Re: 2 gig limits Date: 2/10/2004 12:24:18 PM Eastern Standard Time From: [EMAIL PROTECTED] To: U2 Users Discussion List [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] Sent from the Internet (Details Dave, If those 2GB files are static files then there is a serious problem in your future. If they are dynamic then they can grow beyond 2GB. FILE.STAT 'filename' at the colon prompt will tell you what the file is. -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
The 2GB file size limit is a traditional Unix-based operating system limit. The 2GB file limit appies to *all* files, UniVerse or not. Then maybe I missed something in an earlier post. What is a general suggestion if we expect any sort of file to be larger than 2 gigs? Thanks, JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
It would have to be a 64bit file, there are no exceptions as this is a limitation brought on by the size of a number. (I think a couple of earlier posters had the numbers involved) therefore you literally cannot create a file larger than 2Gb with 32 bit addressing. We have a substantial number of files over 2Gb and haven't had any issue with converting to or using 64bit files. If you expect a file to be 2Gb then just CREATE.FILE FILE.NAME type modulo separation OTHER.PARAMS 64BIT or if it is an existing file RESIZE FILE.NAME new.type new.module new.separation 64BIT will convert it in a trice (or two). Regards David Logan Database Administrator HP Managed Services 139 Frome Street, Adelaide 5000 Australia +61 8 8408 4273 +61 417 268 665 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason Theis Sent: Wednesday, 11 February 2004 7:12 AM To: 'U2 Users Discussion List' Subject: RE: 2 gig limits The 2GB file size limit is a traditional Unix-based operating system limit. The 2GB file limit appies to *all* files, UniVerse or not. Then maybe I missed something in an earlier post. What is a general suggestion if we expect any sort of file to be larger than 2 gigs? Thanks, JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
You can make larger files, but U2 cannot address them, unless you enable 64-bit addressing. The limit is in the unix file /etc/limits (at least on AIX), as fsize. Fsize is usually expressed in 512b chunks, so a) check, and b) figure out your upper size requirement in local block size. Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald From: Jason Theis [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] To: 'U2 Users Discussion List' [EMAIL PROTECTED] Subject: RE: 2 gig limits Date: Tue, 10 Feb 2004 13:41:31 -0700 The 2GB file size limit is a traditional Unix-based operating system limit. The 2GB file limit appies to *all* files, UniVerse or not. Then maybe I missed something in an earlier post. What is a general suggestion if we expect any sort of file to be larger than 2 gigs? Thanks, JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ Let the advanced features services of MSN Internet Software maximize your online time. http://click.atdmt.com/AVE/go/onm00200363ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
The limit is an old one from Unix having 32-bit addressing. On a system with 32-bit addressing, the limit applied to all files, including backup images (at least to disk - I'm not certain about on tape drives, although I'd assume so). Now, we have 64-bit addressing, so the upper limit is in the pentabyte range. UniVerse and Unidata still have a default configuration parameter of 32-bit addressing. This parameter is easily changed to 64. Currently, the cost associated with going to 64-bit addressing for UniVerse Unidata is the loss of a particular tool which is useful in repairing file corruption, filepeek. File corruption is pretty rare, but not unheard of, especially as hardware fails. By going to a scheme like RAID0+1 with transaction logging, you probably won't miss (watch the thread this starts...) filepeek. As an aside, reducing the amount of data in overflow reduces the risk of corruption, by minimizing the number of links, which are failure points. So you can enable U2 64-bit addressing in the (udt/uv)config file, which will then make the limit a historical curiosity. Or you can use dynamic files (although at the most - with a lot of luck - this gives you 4Gb), or in UniVerse you can use distributed files, which imho are a better choice anyway, making the size of a file limited only by your disk drive budget. Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald From: Jason Theis [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] To: 'U2 Users Discussion List' [EMAIL PROTECTED] Subject: RE: 2 gig limits Date: Tue, 10 Feb 2004 12:40:15 -0700 We are looking to move to Universe. Does a 2 gig limit apply to Universe as well? Does it only apply to the backup or live data? JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ Optimize your Internet experience to the max with the new MSN Premium Internet Software. http://click.atdmt.com/AVE/go/onm00200359ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
We only use AIX and possibly (at a later date) Linux. JT -Original Message- From: Geoffrey Mitchell [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 10, 2004 2:41 PM To: U2 Users Discussion List Subject: RE: 2 gig limits Under HP/UX there are some known bugs with 64-bit files in older UniVerse versions which can lead to file truncation. Be sure to research the applicable release for your O/S and make sure there are no known problems before you decide to implement this solution. On Tue, 2004-02-10 at 14:51, Logan, David (SST - Adelaide) wrote: It would have to be a 64bit file, there are no exceptions as this is a limitation brought on by the size of a number. (I think a couple of earlier posters had the numbers involved) therefore you literally cannot create a file larger than 2Gb with 32 bit addressing. We have a substantial number of files over 2Gb and haven't had any issue with converting to or using 64bit files. If you expect a file to be 2Gb then just CREATE.FILE FILE.NAME type modulo separation OTHER.PARAMS 64BIT or if it is an existing file RESIZE FILE.NAME new.type new.module new.separation 64BIT will convert it in a trice (or two). Regards David Logan Database Administrator HP Managed Services 139 Frome Street, Adelaide 5000 Australia +61 8 8408 4273 +61 417 268 665 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason Theis Sent: Wednesday, 11 February 2004 7:12 AM To: 'U2 Users Discussion List' Subject: RE: 2 gig limits The 2GB file size limit is a traditional Unix-based operating system limit. The 2GB file limit appies to *all* files, UniVerse or not. Then maybe I missed something in an earlier post. What is a general suggestion if we expect any sort of file to be larger than 2 gigs? Thanks, JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- Geoffrey Mitchell 314-684-1062 Programmer/Analyst [EMAIL PROTECTED] Knights Direct -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
... On Behalf Of Dan Fitzgerald You can make larger files, but U2 cannot address them, unless you enable 64-bit addressing UV *mostly* handled 2GB files, but I had trouble enabling them for UV's transaction logging. If I remember, UV used a unix utility - maybe fsync? - that was only good for files 2GB. Many unix utilities won't work for files 2GB. I would the list suppose that might vary from platform to platform. I think I had trouble with rcp or cp the other day, too, but I don't remember. Rule of thumb: try to stay under 2GB. Chuck Stevenson -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
Hi Dan, We can use filepeek quite happily on our 64bit files. Do you mean uvfixfile? I know that has restrictions and cannot be used on 64bit files. The parameter in the uvconfig file is 64BIT. If this is set to 1, all files are created by default as 64bit files. Regards David Logan Database Administrator HP Managed Services 139 Frome Street, Adelaide 5000 Australia +61 8 8408 4273 +61 417 268 665 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dan Fitzgerald Sent: Wednesday, 11 February 2004 7:42 AM To: [EMAIL PROTECTED] Subject: RE: 2 gig limits The limit is an old one from Unix having 32-bit addressing. On a system with 32-bit addressing, the limit applied to all files, including backup images (at least to disk - I'm not certain about on tape drives, although I'd assume so). Now, we have 64-bit addressing, so the upper limit is in the pentabyte range. UniVerse and Unidata still have a default configuration parameter of 32-bit addressing. This parameter is easily changed to 64. Currently, the cost associated with going to 64-bit addressing for UniVerse Unidata is the loss of a particular tool which is useful in repairing file corruption, filepeek. File corruption is pretty rare, but not unheard of, especially as hardware fails. By going to a scheme like RAID0+1 with transaction logging, you probably won't miss (watch the thread this starts...) filepeek. As an aside, reducing the amount of data in overflow reduces the risk of corruption, by minimizing the number of links, which are failure points. So you can enable U2 64-bit addressing in the (udt/uv)config file, which will then make the limit a historical curiosity. Or you can use dynamic files (although at the most - with a lot of luck - this gives you 4Gb), or in UniVerse you can use distributed files, which imho are a better choice anyway, making the size of a file limited only by your disk drive budget. Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald From: Jason Theis [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] To: 'U2 Users Discussion List' [EMAIL PROTECTED] Subject: RE: 2 gig limits Date: Tue, 10 Feb 2004 12:40:15 -0700 We are looking to move to Universe. Does a 2 gig limit apply to Universe as well? Does it only apply to the backup or live data? JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ Optimize your Internet experience to the max with the new MSN Premium Internet Software. http://click.atdmt.com/AVE/go/onm00200359ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
Yes; I remembered one of those didn't work, and I chose the wrong one. Shoulda fired up my UV server checked first... Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald From: Logan, David (SST - Adelaide) [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] To: U2 Users Discussion List [EMAIL PROTECTED] Subject: RE: 2 gig limits Date: Wed, 11 Feb 2004 09:42:55 +1100 Hi Dan, We can use filepeek quite happily on our 64bit files. Do you mean uvfixfile? I know that has restrictions and cannot be used on 64bit files. The parameter in the uvconfig file is 64BIT. If this is set to 1, all files are created by default as 64bit files. Regards David Logan Database Administrator HP Managed Services 139 Frome Street, Adelaide 5000 Australia +61 8 8408 4273 +61 417 268 665 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dan Fitzgerald Sent: Wednesday, 11 February 2004 7:42 AM To: [EMAIL PROTECTED] Subject: RE: 2 gig limits The limit is an old one from Unix having 32-bit addressing. On a system with 32-bit addressing, the limit applied to all files, including backup images (at least to disk - I'm not certain about on tape drives, although I'd assume so). Now, we have 64-bit addressing, so the upper limit is in the pentabyte range. UniVerse and Unidata still have a default configuration parameter of 32-bit addressing. This parameter is easily changed to 64. Currently, the cost associated with going to 64-bit addressing for UniVerse Unidata is the loss of a particular tool which is useful in repairing file corruption, filepeek. File corruption is pretty rare, but not unheard of, especially as hardware fails. By going to a scheme like RAID0+1 with transaction logging, you probably won't miss (watch the thread this starts...) filepeek. As an aside, reducing the amount of data in overflow reduces the risk of corruption, by minimizing the number of links, which are failure points. So you can enable U2 64-bit addressing in the (udt/uv)config file, which will then make the limit a historical curiosity. Or you can use dynamic files (although at the most - with a lot of luck - this gives you 4Gb), or in UniVerse you can use distributed files, which imho are a better choice anyway, making the size of a file limited only by your disk drive budget. Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald From: Jason Theis [EMAIL PROTECTED] Reply-To: U2 Users Discussion List [EMAIL PROTECTED] To: 'U2 Users Discussion List' [EMAIL PROTECTED] Subject: RE: 2 gig limits Date: Tue, 10 Feb 2004 12:40:15 -0700 We are looking to move to Universe. Does a 2 gig limit apply to Universe as well? Does it only apply to the backup or live data? JT -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ Optimize your Internet experience to the max with the new MSN Premium Internet Software. http://click.atdmt.com/AVE/go/onm00200359ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ Check out the great features of the new MSN 9 Dial-up, with the MSN Dial-up Accelerator. http://click.atdmt.com/AVE/go/onm00200361ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
From: Dan Fitzgerald [mailto:[EMAIL PROTECTED] You can make larger files, but U2 cannot address them, unless you enable 64-bit addressing. The limit is in the unix file /etc/limits (at least on AIX), as fsize. Fsize is usually expressed in 512b chunks, so a) check, and b) figure out your upper size requirement in local block size. Can Unidata do 64 bit addressing? I thought only Universe that had capability. The answer to the original question is make the file dynamic. That's the quick fix and should handle the file in the long term. - jmh -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
From: Dan Fitzgerald [mailto:[EMAIL PROTECTED] will then make the limit a historical curiosity. Or you can use dynamic files (although at the most - with a lot of luck - this gives you 4Gb), What makes you say this? We have some dynamic files that are well over 10-15 gig. They have multiple dat files that are all well under the 2 gig limit. - jmh -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: 2 gig limits
From: Horn, John From: Dan Fitzgerald Or you can use dynamic files (although at the most - with a lot of luck - this gives you 4Gb), What makes you say this? We have some dynamic files that are well over 10-15 gig. They have multiple dat files that are all well under the 2 gig limit. John, as you noted in a previous post, UniVerse and UniData are different. UniVerse dynamic files allow only a single DATA.30 and a single OVER.30 file, so with 32bit addressing that allows a maximum of 2GB primamry data and 2GB of overflow, but of course it is very hard to organise a file to get close to that distribution of data between data and overflow. In UniData a dynamic file can contain up to 256 part files each of which can be configured up to 2GB if you really want, so there is no practical limit on the amount of data you can store even with 32bit addressing. I suspect Dan was busy thinking about UniVerse when he posted, even though the OP is actually running UniData. Cheers, Ken -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
