Re: [U2] Learning about file sizing
Marc, I need to understand more clearly about the split/merge formula. Below it states: SPLIT = INT(RECORDS PER BLOCK * IDSIZE * 100 / BLOCKSIZE) SPLIT = INT( 9 * 17 * 100 / 1024) = 14 How did they come up with 9 as the RECORDS PER BLOCK from the file status outlined? Sincerely, David Laansma IT Manager Hubbard Supply Co. Direct: 810-342-7143 Office: 810-234-8681 Fax: 810-234-6142 www.hubbardsupply.com "Delivering Products, Services and Innovative Solutions" -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rutherford, Marc Sent: Tuesday, June 05, 2012 3:50 PM To: U2 Users List Subject: Re: [U2] Learning about file sizing Rod, Excellent post! I have a file I have been wanting to convert to dynamic. Since it not something I do every day I have been stalling for a while now... Thanks, Marc Rutherford Principal Programmer Analyst Advanced Bionics LLC 661) 362 1754 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen, Rodney A (Rod) 46K Sent: Tuesday, June 05, 2012 10:53 AM To: 'U2 Users List' Subject: Re: [U2] Learning about file sizing Can't remember if this came from Wally or not a long time ago. But I use it to figure out Split/Merge. I have a development box that has a copy of production that I can play with. So I do a lot of playing with mod and sep and depend on GROUP.STAT to give me some idea of how groups are being populated. Sizing Dynamic Files Technote (FAQ) Problem Sometimes, administrators would like some ideas and insights on how to configure dynamic files to maximize file access speed and minimize the physical size. This article describes one process for making this determination. Solution To improve dynamic file performance an administrator can choose a new modulo and/or block size. Other important factors, however, are the percent standard deviation of the record size, the correct the hash type, and the split load percent. The first step is to generate file statistics using the ECL command FILE.STAT (in ECLTYPE U mode). The percent standard deviation can be obtained by the following formula: "Standard deviation from average" divided by "Average number of bytes in a record". Ideally, this percent would be zero - all records are exactly the same size. Having all records the same size makes calculations more accurate for our file sizing purposes. A standard deviation percent under 15-20% means the variation of record sizes is less than perfect but we can still predict well enough to be confident that the problem has a satisfactory solution. However, it is very common in the U2 world for a file design to have been left in service beyond what is reasonable for today's situations - i.e. what worked well in 1980 may not be a good solution for much larger files than was originally anticipated. So, if in the old days you had, say, 10 multivalues in most records and today you have between 20 and 3000, then it is easy to see how the percent standard deviation for record size can creep up over the years without being noticed. Anyway, we'll slog forward on the assumption that the standard deviation percent is "good". Final point: a high standard deviation percent for record size usually leads to wasted space, either in the form of sparsely populated primary groups and/or excessive overflow. A high standard deviation percent can create a situation where there is no "good" answer. An important factor in correct file sizing is to determine the better hashing algorithm - either type 0 or 1. It is useful to keep an open mind on this because hash type is another thing that can be set correctly and, over time, the format of the ids changes, and now the other hash type is better. First, you should always do ANALYZE.FILE filename and look at the "Keys" column. If you see consistency in the number as you look down the column, then the algorithm currently in place is likely correct. It can be variable enough to warrant further study. How you do this kind of analysis is to select a sample of 10,000 record ids from the file. Then create two dynamic files (one a type 0, the other a type 1) of blocksize 1024 and a modulo of 3. Then, CONFIGURE.FILE to set the MERGE.LOAD to 5 and the SPLIT.LOAD to 10. This configuration helps exagerate the results of the testing to make the decision a little easier. Then, populate each of the files using the sample list of ids and th e empty string for a record. Whichever file is the smaller is usually the better hash type. Determine Id Size and Record Size Get two numbers from the FILE.STAT report: "Average number of bytes in a record"(avg rec size), rounded up to the next whole number; and, "Average number of bytes in record ID"(id size), rounded up to the next whole number. Follow these steps: 1.IDSIZE = id size from report above + 8 2.DATASIZE = avg rec size from report above -
Re: [U2] Regarding: use of I - Descriptor
In SQL Server, I think it is called UDF = user-defined function --dawn On Mon, Jun 25, 2012 at 3:20 PM, Wols Lists wrote: > On 25/06/12 17:47, Oaks, Harold wrote: >> Hi Satya: >> >> An I-descriptor is the same as an I-type (or a V-type). >> >> It's like a little program in the DICT (dictionary) of a file. It can >> do very complex things, but quite often is used for very simple things, >> but those simple things can be quite helpful. Here is an example- > > I *believe* MS SQL-server has a similar facility called a "calculated > field". A colleague told me a story of how he discovered this then went > on a SQL training course. When he tried to explain this to the > instructor and the class, most of them just couldn't understand. But the > few (including the instructor) who "got it" just couldn't get over how > useful this facility could be. > > Cheers, > Wol > ___ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users -- Dawn M. Wolthuis Take and give some delight today ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regarding: use of I - Descriptor
On 25/06/12 17:47, Oaks, Harold wrote: > Hi Satya: > > An I-descriptor is the same as an I-type (or a V-type). > > It's like a little program in the DICT (dictionary) of a file. It can > do very complex things, but quite often is used for very simple things, > but those simple things can be quite helpful. Here is an example- I *believe* MS SQL-server has a similar facility called a "calculated field". A colleague told me a story of how he discovered this then went on a SQL training course. When he tried to explain this to the instructor and the class, most of them just couldn't understand. But the few (including the instructor) who "got it" just couldn't get over how useful this facility could be. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regarding: use of I - Descriptor
Satya, Roughly speaking, and I-Desc is a way to manipulate the data so that you can select or display data in other ways. You can get data from another file, perform some math on multiple fields, concatenate data, call a basic subroutine, or any combination of these. You can have multiple commands within an I-Desc with each piece separated by a semi-colon, and then take action based on each "piece" of the code. You can even put alternate indexes on I-Desc, though this can lead to serious problems if you are indexing a translate. You generally cannot do updates with an I-Desc, though if the I-Desc called a subroutine, you COULD do anything but I have never seen this done (and with very good reason). On a wild tangent to what I just said, I have always thought how easy (in a purest way) that you COULD perform some task like month end by having a file with special record and an I-Desc that DOES run update subroutines (yeah I know, I just said not to do that, but the idea is cool). If set up correctly, you COULD simply run your month end by sorting the special file with the I-Desc. For example: SORT MONTH.END.CTL DO.TASK Where DO.TASK is an I-Desc that calls a subroutine that knows what to do based on the key of each record. Again, NOT a good idea, but the capability is there. John -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of satya satya Sent: Monday, June 25, 2012 12:10 PM To: U2 Users List Subject: [U2] Regarding: use of I - Descriptor Hi, Could any one of you explain What is I - Descriptor and What is the use of it. Thank you, Satya ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regarding: use of I - Descriptor
I'll add a tidbit that some of the similar (not identical) names are: virtual field derived data calculated field computed column user-defined function stored procedure --dawn On Mon, Jun 25, 2012 at 11:47 AM, Oaks, Harold wrote: > Hi Satya: > > An I-descriptor is the same as an I-type (or a V-type). > > It's like a little program in the DICT (dictionary) of a file. It can > do very complex things, but quite often is used for very simple things, > but those simple things can be quite helpful. Here is an example- > > In the DICT of the INMATE file there are D-types for last name, first > name, and middle name, which are LNAME, FNAME, and MNAME. > Now, one can see a listing of inmate names by typing: > > LIST INMATE LNAME FNAME MNAME > > and get this: > > INMATE NAME FIRST NAME MIDDLE NAME > 86144 JOHNSON PHILLIP > 779501 THOMAS ERNEST ROBERT > 192814 SMITH THOMAS DEAN > 204251 TOONSDALE PAUL JAMES > > However, this takes up a lot of space for each name with the three names > combining for 45 characters. To save space, let's create an I-type to > put the last, first, and middle names together as one field: > ED DICT INMATE FULLNAME > This is a Type "I" Descriptor last compiled on 02/08/12 at 15:38. > 0001: I > 0002: LNAME:', ':FNAME:' ':MNAME > 0003: > 0004: FULL NAME > 0005: 25L > 0006: S > > You see in field 2 is that the three parts of the name are concatenated > together, with a comma after the last name. This makes the display much > shorter and works well for most anyone's name: > INMATE FULL NAME... > > 86144 JOHNSON, PHILLIP > 779501 THOMAS, ERNEST ROBERT > 192814 SMITH, THOMAS DEAN > 204251 TOONSDALE, PAUL JAMES > > The field now is only 25 characters in display width, but only a few > person's names will exceed 25 characters and need to wrap to the next > line, so that saves a lot of space. The I-type was very simple to > create and understand. Oh yes, after creating the I-type field in the > DICT, one needs to compile it with: > COMPILE.DICT INMATE FULLNAME > As I said, it's actually a little program. > > These little programs - the I-types - are one of the real strengths of > the multivalue development environment. > > Harold Oaks > Clark County, WA > > > > > -Original Message- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of satya satya > Sent: Monday, June 25, 2012 9:10 AM > To: U2 Users List > Subject: [U2] Regarding: use of I - Descriptor > > Hi, > > Could any one of you explain What is I - Descriptor and What is the use > of it. > > Thank you, > Satya > ___ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > This e-mail and related attachments and any response may be subject to public > disclosure under state law. > ___ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users -- Dawn M. Wolthuis Take and give some delight today ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regarding: use of I - Descriptor
Hi Satya, These are really basic questions. You would be best to take a training course. An I-descriptor is a calculated value based on other dictionary items. Field 2 holds an expression that can include most of the things that you can do on the right hand side of an equals in an assignment statement in Basic. For example, you might have an I-descriptor named PROFIT with an expression of SELL - COST Once you have created the dictionary item, you can use it just like any "real" data field in a query processor command. By far the most important part of I-descriptors is the ability to use the TRANS() function to get data from other tables. Martin Phillips Ladybridge Systems Ltd 17b Coldstream Lane, Hardingstone, Northampton NN4 6DB, England +44 (0)1604-709200 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of satya satya Sent: 25 June 2012 17:10 To: U2 Users List Subject: [U2] Regarding: use of I - Descriptor Hi, Could any one of you explain What is I - Descriptor and What is the use of it. Thank you, Satya ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regarding: use of I - Descriptor
Hi Satya: An I-descriptor is the same as an I-type (or a V-type). It's like a little program in the DICT (dictionary) of a file. It can do very complex things, but quite often is used for very simple things, but those simple things can be quite helpful. Here is an example- In the DICT of the INMATE file there are D-types for last name, first name, and middle name, which are LNAME, FNAME, and MNAME. Now, one can see a listing of inmate names by typing: LIST INMATE LNAME FNAME MNAME and get this: INMATENAMEFIRST NAMEMIDDLE NAME 86144JOHNSON PHILLIP 779501THOMAS ERNESTROBERT 192814SMITHTHOMASDEAN 204251TOONSDALEPAUL JAMES However, this takes up a lot of space for each name with the three names combining for 45 characters. To save space, let's create an I-type to put the last, first, and middle names together as one field: ED DICT INMATE FULLNAME This is a Type "I" Descriptor last compiled on 02/08/12 at 15:38. 0001: I 0002: LNAME:', ':FNAME:' ':MNAME 0003: 0004: FULL NAME 0005: 25L 0006: S You see in field 2 is that the three parts of the name are concatenated together, with a comma after the last name. This makes the display much shorter and works well for most anyone's name: INMATEFULL NAME... 86144JOHNSON, PHILLIP 779501THOMAS, ERNEST ROBERT 192814SMITH, THOMAS DEAN 204251TOONSDALE, PAUL JAMES The field now is only 25 characters in display width, but only a few person's names will exceed 25 characters and need to wrap to the next line, so that saves a lot of space. The I-type was very simple to create and understand. Oh yes, after creating the I-type field in the DICT, one needs to compile it with: COMPILE.DICT INMATE FULLNAME As I said, it's actually a little program. These little programs - the I-types - are one of the real strengths of the multivalue development environment. Harold Oaks Clark County, WA -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of satya satya Sent: Monday, June 25, 2012 9:10 AM To: U2 Users List Subject: [U2] Regarding: use of I - Descriptor Hi, Could any one of you explain What is I - Descriptor and What is the use of it. Thank you, Satya ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users This e-mail and related attachments and any response may be subject to public disclosure under state law. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] UV Unix File Recovery
Brian: How many servers do you handle like this. This sounds a lot like Mozy Pro (that kind of thing). Bill - Original Message - *From:* br...@brianleach.co.uk *To:* 'U2 Users List' *Date:* 6/23/2012 2:23 AM *Subject:* Re: [U2] UV Unix File Recovery If you're on windows, take a look at Syncrify. It's essentially doing an rsync for Windows: I've been using it for a few months now on the client side and it's saved me a load of time for doing offsite backup to a hosted VM for peace of mind. Getting the first backup took me several nights(!) but my ISP doesn't charge for traffic between midnight and 6am and once it had that in place it's perfect for scheduling. Now it rsyncs through about 60GB worth of backups in about half an hour over a slow (rural) connection. In-house on a lan it should be very quick. Remember that whatever route you take, you must pause the database and be on a relatively recent version of UniVerse. Going back, dbpause didn't sync the shared memory headers recording dynamic file loads/splits/pointers so on restoring they would be corrupted. Plus there's always the new replication functionality to consider. From the few reports I've heard it's a lot more solid now. Brian -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wols Lists Sent: 23 June 2012 00:22 To: u2-users@listserver.u2ug.org Subject: Re: [U2] UV Unix File Recovery On 22/06/12 20:13, Bill Haskett wrote: George: Unfortunately, I'm on Windows. I do full backups each day, but the 15Gb backup files shut down the dbms for about 30 minutes each night. We're not a 24/7 shop by any means, but we do span a number of time zones, so our window for backups is about three hours each evening. I've always wanted to use something simple but can't find anything. One would think your backup method (mirrors, breaking them, backing up the mirror, then re-syncing) would be part of the U2 admin guide (or be on some wiki). I do this with a couple of simple Windows scripts, but it's strickly a full backup operation with no mirrors. I did have to change the scripts and the method of implementation for Windows 2008 R2 from previous windows using "ntbackup". Sounds like you want proper mirrored disks on your server. dbpause, break mirror, dbresume. Then you can back up the broken mirror at your leisure before resuming the mirror. Your database won't even be down a moment. As for minor changes to a 2Gb file, that's where btrfs would come in handy. It's a "copy on write" filesystem, so when you change a file it only updates the bits that have changed. And it cascades those changes up the hierarchy, so that if you "snapshot" the file system, it will archive the then-root of the filesystem. All new changes go to a new root. Only thing is, if you want to get back to a previous state of the filesystem (ie retrieve a backup), I understand it's a reboot. But if you had the true mirror on your server, you could run an infinite loop of dbpause, break mirror, dbresume, rsync broken mirror to linux btrfs, snapshot btrfs and restore mirror, wait for mirror to resync, rinse and repeat. It would take an awfully long time to fill up the linux backup server's disk... (rsync is the unix command that will sync two file systems, and it's very good at only updating the parts of files that have changed). Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
[U2] Regarding: use of I - Descriptor
Hi, Could any one of you explain What is I - Descriptor and What is the use of it. Thank you, Satya ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
