Agreed, the first step is to create the file so there are as many DAT files as necessary and only one or two OVER files. Once you've done that, copied all of the records over to the new file and it remains with only one or two OVER files, THEN do some analytics to establish the SPLIT/MERGE and the hash type (KEYONLY vs KEYDATA)
Sincerely, David Laansma Hubbard Supply Co. Direct: 810-342-7143 Office: 810-234-8681 Fax: 810-234-6142 www.hubbardsupply.com "Delivering Products, Services and Innovative Solutions" -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen, Rodney A (Rod) 46K Sent: Friday, July 12, 2013 12:56 PM To: U2 Users List Subject: Re: [U2] UniData Dynamic File Splitting Question We have some very large dynamic files. Some are over 100 gigabytes in size. Too me it is weird that you have one dat portion and 45 overflow portions of the file. My 100 gig file has 43 Dat portions and 1 overflow, which is what I try to maintain. I want as few overflow portions of the file as I can get. What does GROUP.STAT say about this file? When you do the guide, it shows how many records fall into 1-512 512-1028 1028-2056 etc. I try and adjust my block size for the file to get close to 80-90% of the records in the file to fit into the block size. I usually don't use memresize to rebuild my dynamic files. I create a new file with the configuration I want. We have developed a process called PHANTOM.COPY. It requires a list of the id's that exist in the current file. Then if I tell PHANTOM.COPY to run using 10 phantoms, it breaks the list up into 10 pieces and does simultaneous copies. So it is not much slower than memresize. The reasoning for this is that memresize creates one overflow portion for every dat portion. And I don't want my 100 gig file to have 43 dats and 43 overflow portions. I want 43 dats and one overflow if I can do it. And PHANTOM.COPY allows me to get that setup. This allows me to figure out what files need resizing too. If I have more than a couple overflow portions, it is time to rebuild the file again. Are you using KEYONLY or KEYDATA for your dynamic files. We have had problems with KEYDATA when we have a keys that have #''s for separators in the key. So for these files, I use KEYONLY. We use all kinds of dynamic files and most are quite large. We are on 7.1 though. So maybe 7.2 has issues. - Rod -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Cinda Goff Sent: Friday, July 12, 2013 11:35 AM To: u2-users@listserver.u2ug.org Subject: [U2] UniData Dynamic File Splitting Question I work with the North Carolina Community College System which supports the 58 NC colleges. We have an unusual dynamic file issue at a couple of the colleges and am hoping someone could provide some insight/direction. OS - Solaris 10 UniData version 7.2.5 (have upgraded a couple of times in the past from 6.1 and 7.1) Colleges have been running Ellucian's Colleague software since 2001 The problems do not seem to be occurring in all dynamic files but where there are issues, there are more than 100 files affected. From timestamps in some of the files, we can see that this has been going on for several years but we are unaware of the issue until the files have split into so many parts that the application can no longer open enough files to read/write the files. Database tools do not show these files as undersized or in overflow2. Guide will show you the many parts if you know what to look for but it is not obvious that the file is fragmented. When we found the issue at the first college we opened a call with Ellucian and was told to resize the files. We had a call to Rocket opened and unfortunately didn't get much further. The Rocket support analyst looked at udtdiags for us and did not find anything out of the ordinary. Rocket identified a couple of potential dynamic file issues at our UniData version and suggested that we upgraded to UniData 7.3. One of the identified issues seemed to apply but it did not explain all issues. We are looking at the UniData upgrade but we likely will still need to address the current file issues. We started resizing files and the files were compressed as expected but soon began splitting again. To 'fix' the files now, we are manually recreating the files, copying data to new files, renaming everything and rebuilding any indices. Long and tedious process but the file seem to behave better, at least so far. Below is a listing of one of the main application files and all its parts. Many of the files have many more parts that the listing below. The splitting occurs with dat, over and idx with files some files having many of one part while other have many of all. If anyone has any insight into how/why the files split this way and suggestion on how to correct and prevent it in the future, I would very much like to hear your ideas. Thanks for your input and suggestions. Cinda Goff sun2:/datatel/coll18/production/apphome/DATA/DATA_P/PERSON>165-$ ls -al total 533958 drwxrwx--- 2 datatel users 1536 Jun 26 12:42 . drwxrwx--- 269 datatel users 13824 Feb 18 10:24 .. -rw-rw---- 1 datatel users 6292480 Jul 1 12:57 dat001 -rw-rw---- 1 datatel users 152743936 Jul 1 12:56 idx001 -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx002 -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx003 -rw-rw-rw- 1 ijackson users 4096 Aug 12 2011 idx004 -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx005 -rw-rw-rw- 1 sbarnes users 4096 Aug 12 2011 idx006 -rw-rw-rw- 1 whardwic users 4096 Nov 5 2011 idx007 -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx008 -rw-rw-rw- 1 jlevister users 4096 Nov 5 2011 idx009 -rw-rw-rw- 1 rcox users 4096 Nov 5 2011 idx010 -rw-rw-rw- 1 gmoore users 4096 Nov 5 2011 idx011 -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx012 -rw-rw-rw- 1 tbozeman users 4096 Nov 5 2011 idx013 -rw-rw-rw- 1 pperry users 4096 Nov 5 2011 idx014 -rw-rw-rw- 1 kconner users 4096 Nov 5 2011 idx015 -rw-rw-rw- 1 pperry users 4096 Nov 5 2011 idx016 -rw-rw-rw- 1 gmoore users 4096 Nov 5 2011 idx017 -rw-rw-rw- 1 sbarnes users 4096 Nov 5 2011 idx018 -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx019 -rw-rw---- 1 datatel users 113200128 Jul 1 12:57 over001 -rw-rw---- 1 whardwic users 19456 Jun 20 08:26 over002 -rw-rw---- 1 ijackson users 18432 May 16 08:04 over003 -rw-rw---- 1 gmoore users 18432 May 14 12:09 over004 -rw-rw---- 1 sbarnes users 17408 Jun 4 19:03 over005 -rw-rw---- 1 llowry users 18432 Feb 1 13:01 over006 -rw-rw---- 1 dmiadmin users 17408 Jun 20 08:26 over007 -rw-rw---- 1 gmoore users 24576 Jun 28 07:31 over008 -rw-rw---- 1 gmoore users 18432 Mar 14 09:10 over009 -rw-rw---- 1 gmoore users 21504 May 25 06:15 over010 -rw-rw---- 1 gmoore users 18432 Jun 22 12:43 over011 -rw-rw---- 1 ijackson users 19456 Jun 4 20:05 over012 -rw-rw---- 1 sbarnes users 19456 Jun 17 14:47 over013 -rw-rw---- 1 bsampson users 19456 May 26 06:11 over014 -rw-rw---- 1 gmoore users 23552 Jun 25 06:15 over015 -rw-rw---- 1 plocklea users 21504 Jun 24 16:22 over016 -rw-rw---- 1 gmoore users 19456 Jan 31 11:31 over017 -rw-rw---- 1 gmoore users 22528 Jun 6 12:15 over018 -rw-rw---- 1 sbarnes users 19456 Jun 26 06:15 over019 -rw-rw---- 1 llowry users 21504 Jun 20 08:26 over020 -rw-rw---- 1 pperry users 21504 Jul 1 10:27 over021 -rw-rw---- 1 sbarnes users 19456 Jun 20 15:34 over022 -rw-rw---- 1 ebarton users 17408 Jun 20 08:26 over023 -rw-rw---- 1 pperry users 19456 May 26 06:09 over024 -rw-rw---- 1 mhester users 20480 Jun 12 15:07 over025 -rw-rw---- 1 pperry users 18432 May 20 12:20 over026 -rw-rw---- 1 pperry users 18432 May 6 13:24 over027 -rw-rw---- 1 jhickman users 19456 May 31 12:15 over028 -rw-rw---- 1 mhester users 18432 Jun 26 06:15 over029 -rw-rw---- 1 tbozeman users 19456 May 31 12:15 over030 -rw-rw---- 1 whardwic users 20480 Aug 8 2012 over031 -rw-rw---- 1 dbrooks users 21504 May 31 12:15 over032 -rw-rw---- 1 sbarnes users 21504 May 25 06:14 over033 -rw-rw---- 1 kconner users 20480 May 26 06:14 over034 -rw-rw---- 1 gmoore users 22528 May 26 06:14 over035 -rw-rw---- 1 whardwic users 19456 Jun 20 08:26 over036 -rw-rw---- 1 sbarnes users 23552 Jun 20 08:26 over037 -rw-rw---- 1 mchavis users 20480 Jun 12 15:06 over038 -rw-rw---- 1 sbarnes users 17408 Jun 20 16:28 over039 -rw-rw---- 1 suchavis users 20480 Jun 17 13:23 over040 -rw-rw---- 1 bj0663827 users 20480 Jun 27 06:15 over041 -rw-rw---- 1 bj0663827 users 17408 Apr 9 11:11 over042 -rw-rw---- 1 tsampson users 21504 Jun 3 15:54 over043 -rw-rw---- 1 whardwic users 19456 Jun 20 12:47 over044 -rw-rw---- 1 sbarnes users 21504 Jun 26 12:42 over045 sun2:/datatel/coll18/production/apphome/DATA/DATA_P/PERSON>166-$ -------------------------------------------------- Cinda Goff N.C. Community College System Database Administrator 919 807-7060 vRoom Link: https://sas.elluminate.com/m.jnlp?password=M.BDCC127B096D131E11EAC16A0F9473&sid=2008362 E-mail correspondence to and from this address may be subject to the North Carolina Public Records Law and shall be disclosed to third parties when required by the statutes. (NCGS.Ch.132) _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ------------------------------------------------------------------------------ CONFIDENTIALITY NOTICE: If you have received this email in error, please immediately notify the sender by e-mail at the address shown. This email transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity to whom it is intended even if addressed incorrectly. Please delete it from your files if you are not the intended recipient. Thank you for your compliance. Copyright (c) 2013 Cigna ============================================================================== _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
