[U2] File type 30 (dynamic)
I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
Dynamic files work well in most scenarios. We moved most our files to be DYNAMIC and easy to maintain. There could be a difference in speed but I don't believe is a game stopper. RESIZE it when purged to remove unnecessary space. Cheers! HP On Tue, Jul 9, 2013 at 12:15 PM, Scott Zachary szach...@gardensalive.comwrote: I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- *hp* ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
Your mileage may vary, but almost *all* of our files are type 30's. Some of them grow and are purged annually as you suggest, for example G/L related files. *And* we have never resized them. Not ever. Never. I don't really know why they never need to be resized, some of them have 100,000 records, some have 1000. Maybe they should be resized, maybe it's not efficient. All I know is we never do it. -Original Message- From: Scott Zachary szach...@gardensalive.com To: u2-users u2-users@listserver.u2ug.org Sent: Tue, Jul 9, 2013 9:15 am Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
Once you start splitting groups, there's overhead on every write where that occurs, possibly creating a bottleneck. Dynamic files are good for files with non-consistent item sizes, and also for files with oversized items, but that's not the case here. I'd go with a wide shallow static file for this purpose, even if you had to make it distributed to account for size. Since most of what you'll do with this file is write to it, I wouldn't worry too much about empty groups. Date: Tue, 9 Jul 2013 09:15:20 -0700 From: szach...@gardensalive.com To: u2-users@listserver.u2ug.org Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
For what it is worth: Glad to hear that someone else mostly uses dynamic files. There seems to be a bit of prejudice against them by ex-Pickers, (and a prejudice for them from on PI'ers like me!) I never resize them. On the one hand, they are probably a little slower in the worst case, but they should never be a lot slower, as a statically hashed file can be if it is overloaded. Dynamic files are a bit more susceptible to problems after a system crash, but even that should not be a big problem for a log file. Depending on the volume of logging information, I would suggest the following: Create the file with a MINIMUM.MODULUS option that is 130% of your estimated data / 2000. This will pre-allocate space and give you performance approximating a statically hashed file (slower SELECTS than a new, empty dynamic file too). Before the first time you purge the file, consider using CONFIGURE.FILE to reset the MIMIMUM.MODULUS to the current mod prior to the purge. You are going to reuse that space anyway, so reconfiguring it to not shrink will speed up the purge and future adds. (rhetorically) Doesn't this make it just like a statically hashed file? Sort of, there are internal differences that handle large records better, and you are protected from the bane of an undersized static file. Also, consider using a timestamp instead of a sequential id. Why pound on that NEXT.AVAILABLE record when you are going to need a date and time anyway. There are options in the SYSTEM() function to get the GMT epoch value and Milliseconds. Just increment the milliseconds until you can lock a new record in the file. You will need to do this as a separate value from the GMT seconds, due to numerical accuracy. -Rick -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Tuesday, July 09, 2013 10:10 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] File type 30 (dynamic) Your mileage may vary, but almost *all* of our files are type 30's. Some of them grow and are purged annually as you suggest, for example G/L related files. *And* we have never resized them. Not ever. Never. I don't really know why they never need to be resized, some of them have 100,000 records, some have 1000. Maybe they should be resized, maybe it's not efficient. All I know is we never do it. -Original Message- From: Scott Zachary szach...@gardensalive.com To: u2-users u2-users@listserver.u2ug.org Sent: Tue, Jul 9, 2013 9:15 am Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
I would probably use a distributed file, with a key based on some date elemen, e.g. month. That way you can add partfiles as a month happens and each file would be a static file of a standard size optimized for the number and size of records expected for that month. You can then have a rolling 12-month window without the disruption of having to add/delete records during the same timeframe as you would for purging a dynamic file. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan Fitzgerald Sent: Wednesday, 10 July 2013 6:01 a.m. To: U2 Users List Subject: Re: [U2] File type 30 (dynamic) Once you start splitting groups, there's overhead on every write where that occurs, possibly creating a bottleneck. Dynamic files are good for files with non-consistent item sizes, and also for files with oversized items, but that's not the case here. I'd go with a wide shallow static file for this purpose, even if you had to make it distributed to account for size. Since most of what you'll do with this file is write to it, I wouldn't worry too much about empty groups. Date: Tue, 9 Jul 2013 09:15:20 -0700 From: szach...@gardensalive.com To: u2-users@listserver.u2ug.org Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic- tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
For me, the key point is knowing how many records per day are going to be written to the log. If it's a reasonably small amount, better to write out to a text file with the log date as the key, makes for easy log reading. If it's a bit more than that I agree more or less with Phil below. I would create dynamic subfiles for each month for easy maintenance. The use minimum modulus is ok but I think we're making a big deal about something that might not even be a performance issue. Once again, the important missing data here is knowing how many records are to be written. -Mensaje original- De: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] En nombre de Phil Walker Enviado el: martes, 09 de julio de 2013 15:51 Para: U2 Users List Asunto: Re: [U2] File type 30 (dynamic) I would probably use a distributed file, with a key based on some date elemen, e.g. month. That way you can add partfiles as a month happens and each file would be a static file of a standard size optimized for the number and size of records expected for that month. You can then have a rolling 12-month window without the disruption of having to add/delete records during the same timeframe as you would for purging a dynamic file. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan Fitzgerald Sent: Wednesday, 10 July 2013 6:01 a.m. To: U2 Users List Subject: Re: [U2] File type 30 (dynamic) Once you start splitting groups, there's overhead on every write where that occurs, possibly creating a bottleneck. Dynamic files are good for files with non-consistent item sizes, and also for files with oversized items, but that's not the case here. I'd go with a wide shallow static file for this purpose, even if you had to make it distributed to account for size. Since most of what you'll do with this file is write to it, I wouldn't worry too much about empty groups. Date: Tue, 9 Jul 2013 09:15:20 -0700 From: szach...@gardensalive.com To: u2-users@listserver.u2ug.org Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic- tp41129.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
Scott The idea behind a dynamic file is to spread the pain of administration over the lifetime of an application. So yes, there are overheads and whilst a well-sized static file will outperform it, a dynamic file in turn will normally outperform a badly sized static file. There are some caveats, mostly relating to accessing the file outside of the database. If you use an OS level backup or snapshot, or anything that looks at these from the OS level, you need to make sure your dynamic files are closed OR you have paused your database - Universe holds runtime dynamic file parameters in shared memory and without those getting written back to the file header (which happens in the pause) the files will break if you restore them. Most of the prejudice against them goes back to the old PICKies who migrated to Universe at a time when they weren't particularly stable and had performance issues around concurrency with the way they effectively single threaded their sizing operations. They were also doubly expensive on UNIXes that had tight limitations on the number of open (OS level) files. But that was a long time ago. Brian -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Scott Zachary Sent: 09 July 2013 17:15 To: u2-users@listserver.u2ug.org Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp4112 9.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] File type 30 (dynamic)
The other option is to use something like syslog (you did not say your OS I believe) and then use something like splunk to query the logs, a great tool that you can customise for dashboarding, searching etc. That way Linux will administer the maintenance of them. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Brian Leach Sent: Wednesday, 10 July 2013 7:54 a.m. To: 'U2 Users List' Subject: Re: [U2] File type 30 (dynamic) Scott The idea behind a dynamic file is to spread the pain of administration over the lifetime of an application. So yes, there are overheads and whilst a well-sized static file will outperform it, a dynamic file in turn will normally outperform a badly sized static file. There are some caveats, mostly relating to accessing the file outside of the database. If you use an OS level backup or snapshot, or anything that looks at these from the OS level, you need to make sure your dynamic files are closed OR you have paused your database - Universe holds runtime dynamic file parameters in shared memory and without those getting written back to the file header (which happens in the pause) the files will break if you restore them. Most of the prejudice against them goes back to the old PICKies who migrated to Universe at a time when they weren't particularly stable and had performance issues around concurrency with the way they effectively single threaded their sizing operations. They were also doubly expensive on UNIXes that had tight limitations on the number of open (OS level) files. But that was a long time ago. Brian -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Scott Zachary Sent: 09 July 2013 17:15 To: u2-users@listserver.u2ug.org Subject: [U2] File type 30 (dynamic) I have very little experience with using dynamic (type 30) files and I'm looking for some pointers in that regard. I have a file that I am considering creating as a dynamic file. The file will be a log file with sequential numeric keys. The record size will normally range from about 200 to 700 bytes, with a probable maximum size of less than 1,000 bytes. We expect the file to continually grow without deletions and we will likely purge it annually. What are the pro's and con's of creating this file as type 30 verses other file types? Thanks, Scott Zachary UniVerse Developer Gardens Alive! Inc -- View this message in context: http://u2-universe-unidata.1073795.n5.nabble.com/File-type-30-dynamic-tp4112 9.html Sent from the U2 - Users mailing list archive at Nabble.com. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
