I agree with Dave, SPLIT/MERGE ratios need to be checked.
On Fri, Jul 12, 2013 at 11:00 AM, Dave Laansma <[email protected]>wrote: > Agreed, the first step is to create the file so there are as many DAT > files as necessary and only one or two OVER files. Once you've done that, > copied all of the records over to the new file and it remains with only one > or two OVER files, THEN do some analytics to establish the SPLIT/MERGE and > the hash type (KEYONLY vs KEYDATA) > > Sincerely, > David Laansma > Hubbard Supply Co. > Direct: 810-342-7143 > Office: 810-234-8681 > Fax: 810-234-6142 > www.hubbardsupply.com > "Delivering Products, Services and Innovative Solutions" > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Baakkonen, Rodney A > (Rod) 46K > Sent: Friday, July 12, 2013 12:56 PM > To: U2 Users List > Subject: Re: [U2] UniData Dynamic File Splitting Question > > We have some very large dynamic files. Some are over 100 gigabytes in > size. Too me it is weird that you have one dat portion and 45 overflow > portions of the file. My 100 gig file has 43 Dat portions and 1 overflow, > which is what I try to maintain. I want as few overflow portions of the > file as I can get. > > What does GROUP.STAT say about this file? > > When you do the guide, it shows how many records fall into 1-512 > 512-10281028-2056 etc. I try and adjust my block size for the file to get > close to > 80-90% of the records in the file to fit into the block size. > > I usually don't use memresize to rebuild my dynamic files. I create a new > file with the configuration I want. We have developed a process called > PHANTOM.COPY. It requires a list of the id's that exist in the current > file. Then if I tell PHANTOM.COPY to run using 10 phantoms, it breaks the > list up into 10 pieces and does simultaneous copies. So it is not much > slower than memresize. The reasoning for this is that memresize creates one > overflow portion for every dat portion. And I don't want my 100 gig file to > have 43 dats and 43 overflow portions. I want 43 dats and one overflow if I > can do it. And PHANTOM.COPY allows me to get that setup. This allows me to > figure out what files need resizing too. If I have more than a couple > overflow portions, it is time to rebuild the file again. > > Are you using KEYONLY or KEYDATA for your dynamic files. We have had > problems with KEYDATA when we have a keys that have #''s for separators in > the key. So for these files, I use KEYONLY. > > We use all kinds of dynamic files and most are quite large. We are on 7.1 > though. So maybe 7.2 has issues. - Rod > > > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Cinda Goff > Sent: Friday, July 12, 2013 11:35 AM > To: [email protected] > Subject: [U2] UniData Dynamic File Splitting Question > > I work with the North Carolina Community College System which supports the > 58 NC colleges. We have an unusual dynamic file issue at a couple of the > colleges and am hoping someone could provide some insight/direction. > > OS - Solaris 10 > UniData version 7.2.5 (have upgraded a couple of times in the past from > 6.1 and 7.1) Colleges have been running Ellucian's Colleague software since > 2001 > > The problems do not seem to be occurring in all dynamic files but where > there are issues, there are more than 100 files affected. From timestamps > in some of the files, we can see that this has been going on for several > years but we are unaware of the issue until the files have split into so > many parts that the application can no longer open enough files to > read/write the files. Database tools do not show these files as > undersized or in overflow2. Guide will show you the many parts if you know > what to look for but it is not obvious that the file is fragmented. > > When we found the issue at the first college we opened a call with > Ellucian and was told to resize the files. We had a call to Rocket > opened and unfortunately didn't get much further. The Rocket support > analyst looked at udtdiags for us and did not find anything out of the > ordinary. Rocket identified a couple of potential dynamic file issues at > our UniData version and suggested that we upgraded to UniData 7.3. One of > the identified issues seemed to apply but it did not explain all issues. > We are looking at the UniData upgrade but we likely will still need to > address the current file issues. > > We started resizing files and the files were compressed as expected but > soon began splitting again. To 'fix' the files now, we are manually > recreating the files, copying data to new files, renaming everything and > rebuilding any indices. Long and tedious process but the file seem to > behave better, at least so far. > > Below is a listing of one of the main application files and all its parts. > Many of the files have many more parts that the listing below. The > splitting occurs with dat, over and idx with files some files having many > of one part while other have many of all. If anyone has any insight into > how/why the files split this way and suggestion on how to correct and > prevent it in the future, I would very much like to hear your ideas. > > Thanks for your input and suggestions. > Cinda Goff > > sun2:/datatel/coll18/production/apphome/DATA/DATA_P/PERSON>165-$ ls -al > total 533958 > drwxrwx--- 2 datatel users 1536 Jun 26 12:42 . > drwxrwx--- 269 datatel users 13824 Feb 18 10:24 .. > -rw-rw---- 1 datatel users 6292480 Jul 1 12:57 dat001 > -rw-rw---- 1 datatel users 152743936 Jul 1 12:56 idx001 > -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx002 > -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx003 > -rw-rw-rw- 1 ijackson users 4096 Aug 12 2011 idx004 > -rw-rw-rw- 1 gmoore users 4096 Aug 12 2011 idx005 > -rw-rw-rw- 1 sbarnes users 4096 Aug 12 2011 idx006 > -rw-rw-rw- 1 whardwic users 4096 Nov 5 2011 idx007 > -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx008 > -rw-rw-rw- 1 jlevister users 4096 Nov 5 2011 idx009 > -rw-rw-rw- 1 rcox users 4096 Nov 5 2011 idx010 > -rw-rw-rw- 1 gmoore users 4096 Nov 5 2011 idx011 > -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx012 > -rw-rw-rw- 1 tbozeman users 4096 Nov 5 2011 idx013 > -rw-rw-rw- 1 pperry users 4096 Nov 5 2011 idx014 > -rw-rw-rw- 1 kconner users 4096 Nov 5 2011 idx015 > -rw-rw-rw- 1 pperry users 4096 Nov 5 2011 idx016 > -rw-rw-rw- 1 gmoore users 4096 Nov 5 2011 idx017 > -rw-rw-rw- 1 sbarnes users 4096 Nov 5 2011 idx018 > -rw-rw-rw- 1 jjacobs users 4096 Nov 5 2011 idx019 > -rw-rw---- 1 datatel users 113200128 Jul 1 12:57 over001 > -rw-rw---- 1 whardwic users 19456 Jun 20 08:26 over002 > -rw-rw---- 1 ijackson users 18432 May 16 08:04 over003 > -rw-rw---- 1 gmoore users 18432 May 14 12:09 over004 > -rw-rw---- 1 sbarnes users 17408 Jun 4 19:03 over005 > -rw-rw---- 1 llowry users 18432 Feb 1 13:01 over006 > -rw-rw---- 1 dmiadmin users 17408 Jun 20 08:26 over007 > -rw-rw---- 1 gmoore users 24576 Jun 28 07:31 over008 > -rw-rw---- 1 gmoore users 18432 Mar 14 09:10 over009 > -rw-rw---- 1 gmoore users 21504 May 25 06:15 over010 > -rw-rw---- 1 gmoore users 18432 Jun 22 12:43 over011 > -rw-rw---- 1 ijackson users 19456 Jun 4 20:05 over012 > -rw-rw---- 1 sbarnes users 19456 Jun 17 14:47 over013 > -rw-rw---- 1 bsampson users 19456 May 26 06:11 over014 > -rw-rw---- 1 gmoore users 23552 Jun 25 06:15 over015 > -rw-rw---- 1 plocklea users 21504 Jun 24 16:22 over016 > -rw-rw---- 1 gmoore users 19456 Jan 31 11:31 over017 > -rw-rw---- 1 gmoore users 22528 Jun 6 12:15 over018 > -rw-rw---- 1 sbarnes users 19456 Jun 26 06:15 over019 > -rw-rw---- 1 llowry users 21504 Jun 20 08:26 over020 > -rw-rw---- 1 pperry users 21504 Jul 1 10:27 over021 > -rw-rw---- 1 sbarnes users 19456 Jun 20 15:34 over022 > -rw-rw---- 1 ebarton users 17408 Jun 20 08:26 over023 > -rw-rw---- 1 pperry users 19456 May 26 06:09 over024 > -rw-rw---- 1 mhester users 20480 Jun 12 15:07 over025 > -rw-rw---- 1 pperry users 18432 May 20 12:20 over026 > -rw-rw---- 1 pperry users 18432 May 6 13:24 over027 > -rw-rw---- 1 jhickman users 19456 May 31 12:15 over028 > -rw-rw---- 1 mhester users 18432 Jun 26 06:15 over029 > -rw-rw---- 1 tbozeman users 19456 May 31 12:15 over030 > -rw-rw---- 1 whardwic users 20480 Aug 8 2012 over031 > -rw-rw---- 1 dbrooks users 21504 May 31 12:15 over032 > -rw-rw---- 1 sbarnes users 21504 May 25 06:14 over033 > -rw-rw---- 1 kconner users 20480 May 26 06:14 over034 > -rw-rw---- 1 gmoore users 22528 May 26 06:14 over035 > -rw-rw---- 1 whardwic users 19456 Jun 20 08:26 over036 > -rw-rw---- 1 sbarnes users 23552 Jun 20 08:26 over037 > -rw-rw---- 1 mchavis users 20480 Jun 12 15:06 over038 > -rw-rw---- 1 sbarnes users 17408 Jun 20 16:28 over039 > -rw-rw---- 1 suchavis users 20480 Jun 17 13:23 over040 > -rw-rw---- 1 bj0663827 users 20480 Jun 27 06:15 over041 > -rw-rw---- 1 bj0663827 users 17408 Apr 9 11:11 over042 > -rw-rw---- 1 tsampson users 21504 Jun 3 15:54 over043 > -rw-rw---- 1 whardwic users 19456 Jun 20 12:47 over044 > -rw-rw---- 1 sbarnes users 21504 Jun 26 12:42 over045 > sun2:/datatel/coll18/production/apphome/DATA/DATA_P/PERSON>166-$ > > -------------------------------------------------- > Cinda Goff > N.C. Community College System > Database Administrator > 919 807-7060 > vRoom Link: > https://sas.elluminate.com/m.jnlp?password=M.BDCC127B096D131E11EAC16A0F9473&sid=2008362 > > E-mail correspondence to and from this address may be subject to the North > Carolina Public Records Law and shall be disclosed to third parties when > required by the statutes. (NCGS.Ch.132) > > > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > > ------------------------------------------------------------------------------ > CONFIDENTIALITY NOTICE: If you have received this email in error, please > immediately notify the sender by e-mail at the address shown. > This email transmission may contain confidential information. This > information is intended only for the use of the individual(s) or entity to > whom it is intended even if addressed incorrectly. Please delete it from > your files if you are not the intended recipient. Thank you for your > compliance. Copyright (c) 2013 Cigna > ============================================================================== > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
