Re: [UV/AIX] gzip data to file?
>sh - c 'uv prog | gzip -c9 > file' A disadvantage of this approach is that an run time errors get merged with your data and how do you know if prog runs to completion? >Point taken, however, I'm an optimist and not to mention, we paid a giddy >sum for machines and networks with boat loads of disk space and impressively >marketed reliability features ;-) But writing 10 MB to disk is a problem? Craig -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Hehehe! I like that attitude. Stuart, next you'll be telling us that you write optimistic code too...who needs error checking and handling?!?! LOL Experience makes you wary and somewhat cynical! regards, David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 4:20 PM To: U2 Users Discussion List Subject: RE: [UV/AIX] gzip data to file? > David has a point. ... So if the ftp fails, the file still exists. Point taken, however, I'm an optimist and not to mention, we paid a giddy sum for machines and networks with boat loads of disk space and impressively marketed reliability features ;-) -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
> David has a point. ... So if the ftp fails, the file still exists. Point taken, however, I'm an optimist and not to mention, we paid a giddy sum for machines and networks with boat loads of disk space and impressively marketed reliability features ;-) ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Stuart David has a point. What we do is a writeseq to a disk file and once this is completed the UVBasic program executes a unix routine to do an ftp of the file to another position. So if the ftp fails, the file still exists. The BASIC Program also deletes the previous file as a process of self maintenance Regards David Jordan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hona, David S Sent: Friday, 6 February 2004 2:32 PM To: 'U2 Users Discussion List' Subject: RE: [UV/AIX] gzip data to file? Stuart, Ah, one disadvantage, of doing the 'all-in-one' approach is that if one step fails, you have to start again. A real problem if the output takes hours to produce and it is part of database update phase too. :-( For example, if your compression step failed, due to disk space running out - will you have to run everything again? Our sequential file generation can be re-run also, as it doesn't do any updating itself. This is handy also, in case there is a data issue. Nothing worse that having to restore data, just to you start again. We're experienced about every problem you can think of, hence we're broken up in to 'repeatable' steps. Works for us. Regards, David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 1:06 PM To: U2 Users Discussion List Subject: RE: [UV/AIX] gzip data to file? > If you can, allow the spooler subsystem to do the work. Cheers Ray, that seems like an efficient way to do it. > We create a 80MB sequential file (plus a few small ones) with a UV > BASIC program. Thanks David, I'm thinking something similar but that outputing to a device or spooler with a processing script might be the way to go for us. I'm trying to keep the IO to a minimum if possible. At least if the spooler is doing any it's at a lower (faster?) level than writeseq. > why the aversion to writing the data to disk? I don't think there's a need to write it until the data is in its final shape. The system already has enough IO to deal with and in theory should be able to keep the data in memory, which may or may not use disk caching, but which should be a lighter touch on the system. S -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Stuart, Ah, one disadvantage, of doing the 'all-in-one' approach is that if one step fails, you have to start again. A real problem if the output takes hours to produce and it is part of database update phase too. :-( For example, if your compression step failed, due to disk space running out - will you have to run everything again? Our sequential file generation can be re-run also, as it doesn't do any updating itself. This is handy also, in case there is a data issue. Nothing worse that having to restore data, just to you start again. We're experienced about every problem you can think of, hence we're broken up in to 'repeatable' steps. Works for us. Regards, David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 1:06 PM To: U2 Users Discussion List Subject: RE: [UV/AIX] gzip data to file? > If you can, allow the spooler subsystem to do the work. Cheers Ray, that seems like an efficient way to do it. > We create a 80MB sequential file (plus a few small ones) with a UV BASIC program. Thanks David, I'm thinking something similar but that outputing to a device or spooler with a processing script might be the way to go for us. I'm trying to keep the IO to a minimum if possible. At least if the spooler is doing any it's at a lower (faster?) level than writeseq. > why the aversion to writing the data to disk? I don't think there's a need to write it until the data is in its final shape. The system already has enough IO to deal with and in theory should be able to keep the data in memory, which may or may not use disk caching, but which should be a lighter touch on the system. S -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Hi David (sounds familiar doesn't it!) Perhaps if you were to try zip instead of gzip, as this has a slightly different methodology, that might cure the issue with the unzip on windows. You can get it from http://aixpdslib.seas.ucla.edu/aixpdslib.html Regards David Logan Database Administrator HP Managed Services 139 Frome Street, Adelaide 5000 Australia +61 8 8408 4273 +61 417 268 665 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of djordan Sent: Friday, 6 February 2004 1:25 PM To: 'U2 Users Discussion List' Subject: RE: [UV/AIX] gzip data to file? Hi David Have you found gzip from AIX reliable. I have one client who would gzip the universe sequential file (30MB) and then unzip it on a Win NT4 server and we had a high level of corruption. They don't compress and they transmitt the files in the evening now. I don't know if this was the gzip utility or the Microsoft unzip, although I tried several zip utitlities on Windows and they all ran into the same problem. Regards David Jordan Dacono Holdings Pty Ltd -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hona, David S Sent: Friday, 6 February 2004 12:24 PM To: 'U2 Users Discussion List' Subject: RE: [UV/AIX] gzip data to file? Stuart, We do something similar, but it involves file transfers from a UNIX server to a mainframe. Which then transfers the file to yet another mainframe. Finally, the second mainfame sends it to UNIX server. Don't ask me why! But there are firewalls, third-party vendors involved, etc., etc. We create a 80MB sequential file (plus a few small ones) with a UV BASIC program. Using a shell script: - copy the above file, plus some others into a staging directory - the files are individually compressed using standard UNIX compress (to keep it simple, the third-party didn't or want to use gzip) - all the files are archived into a standard 'tar' archive utility - to disk (again, for simplicity) - the 'tar' file is then transferred to the mainframe via another external shell script - email notifications are sent to end-users to tell them the file was sent to the third-party - sequential files are retained, so we can manually do the file transfer again, if required Also, the second mainframe (the third-party) sends back an empty ASCII file, acknowledging receipt of original file. We use this for an audit trail, of sorts. 'tar' was used, so we didn't have to do individual file transfers. It was a vendor requirement to try to keep the files to together. All of this is triggered from EOM or ad-hoc batch jobs. It works well (err, okay so it took a while to get it there!). Hope this helps. Regards David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 12:01 PM To: U2-Users Subject: [UV/AIX] gzip data to file? [UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped files, the files then being put on a web server for collection. I'm just wondering what the best approach to this might be? How would or have other people approached this? I'm trying to avoid writing the data until after it has been zipped. Rough ideas being... sh - c 'uv prog | gzip -c9 > file' Print to a script? Write to a device (or pipe?)? Output to a java widget? Other? Regards, Stuart Boydell ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Hi David Have you found gzip from AIX reliable. I have one client who would gzip the universe sequential file (30MB) and then unzip it on a Win NT4 server and we had a high level of corruption. They don't compress and they transmitt the files in the evening now. I don't know if this was the gzip utility or the Microsoft unzip, although I tried several zip utitlities on Windows and they all ran into the same problem. Regards David Jordan Dacono Holdings Pty Ltd -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hona, David S Sent: Friday, 6 February 2004 12:24 PM To: 'U2 Users Discussion List' Subject: RE: [UV/AIX] gzip data to file? Stuart, We do something similar, but it involves file transfers from a UNIX server to a mainframe. Which then transfers the file to yet another mainframe. Finally, the second mainfame sends it to UNIX server. Don't ask me why! But there are firewalls, third-party vendors involved, etc., etc. We create a 80MB sequential file (plus a few small ones) with a UV BASIC program. Using a shell script: - copy the above file, plus some others into a staging directory - the files are individually compressed using standard UNIX compress (to keep it simple, the third-party didn't or want to use gzip) - all the files are archived into a standard 'tar' archive utility - to disk (again, for simplicity) - the 'tar' file is then transferred to the mainframe via another external shell script - email notifications are sent to end-users to tell them the file was sent to the third-party - sequential files are retained, so we can manually do the file transfer again, if required Also, the second mainframe (the third-party) sends back an empty ASCII file, acknowledging receipt of original file. We use this for an audit trail, of sorts. 'tar' was used, so we didn't have to do individual file transfers. It was a vendor requirement to try to keep the files to together. All of this is triggered from EOM or ad-hoc batch jobs. It works well (err, okay so it took a while to get it there!). Hope this helps. Regards David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 12:01 PM To: U2-Users Subject: [UV/AIX] gzip data to file? [UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped files, the files then being put on a web server for collection. I'm just wondering what the best approach to this might be? How would or have other people approached this? I'm trying to avoid writing the data until after it has been zipped. Rough ideas being... sh - c 'uv prog | gzip -c9 > file' Print to a script? Write to a device (or pipe?)? Output to a java widget? Other? Regards, Stuart Boydell ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
> If you can, allow the spooler subsystem to do the work. Cheers Ray, that seems like an efficient way to do it. > We create a 80MB sequential file (plus a few small ones) with a UV BASIC program. Thanks David, I'm thinking something similar but that outputing to a device or spooler with a processing script might be the way to go for us. I'm trying to keep the IO to a minimum if possible. At least if the spooler is doing any it's at a lower (faster?) level than writeseq. > why the aversion to writing the data to disk? I don't think there's a need to write it until the data is in its final shape. The system already has enough IO to deal with and in theory should be able to keep the data in memory, which may or may not use disk caching, but which should be a lighter touch on the system. S ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: [UV/AIX] gzip data to file?
>[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped >files, the files then being put on a web server for collection. I'm just >wondering what the best approach to this might be? How would or have other >people approached this? I'm trying to avoid writing the data until after it >has been zipped. Stuart, why the aversion to writing the data to disk? Write the data to a type 19 file and then for each file execute the shell command "gzip FILE" This will cause gzip replace the original file with the compressed data FILE.gz. I think your shell example will work (and it would be my preffered option for thos you suggest) but my suggestion may be simpler to maintain. Craig -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: [UV/AIX] gzip data to file?
If you can, allow the spooler subsystem to do the work. Spool the data, and write a script for the spooler to execute. Compress the data and ftp where you wish it to go. Remember that the spooler runs as root, so you may have some NFS issues should you attempt to do a "cp" to an nfs mounted partition. Regards, Ray D - Original Message - From: "Stuart Boydell" <[EMAIL PROTECTED]> To: "U2-Users" <[EMAIL PROTECTED]> Sent: Thursday, February 05, 2004 5:00 PM Subject: [UV/AIX] gzip data to file? > [UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped > files, the files then being put on a web server for collection. I'm just > wondering what the best approach to this might be? How would or have other > people approached this? I'm trying to avoid writing the data until after it > has been zipped. > > Rough ideas being... > sh - c 'uv prog | gzip -c9 > file' > Print to a script? > Write to a device (or pipe?)? > Output to a java widget? > Other? > > Regards, > Stuart Boydell > > > > > > > > > > > > > > > > > > > ** > This email message and any files transmitted with it are confidential > and intended solely for the use of addressed recipient(s). If you have > received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. > > This footnote also confirms that this email message has been scanned > for the presence of computer viruses. > ** > > -- > u2-users mailing list > [EMAIL PROTECTED] > http://www.oliver.com/mailman/listinfo/u2-users > -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: [UV/AIX] gzip data to file?
Stuart, We do something similar, but it involves file transfers from a UNIX server to a mainframe. Which then transfers the file to yet another mainframe. Finally, the second mainfame sends it to UNIX server. Don't ask me why! But there are firewalls, third-party vendors involved, etc., etc. We create a 80MB sequential file (plus a few small ones) with a UV BASIC program. Using a shell script: - copy the above file, plus some others into a staging directory - the files are individually compressed using standard UNIX compress (to keep it simple, the third-party didn't or want to use gzip) - all the files are archived into a standard 'tar' archive utility - to disk (again, for simplicity) - the 'tar' file is then transferred to the mainframe via another external shell script - email notifications are sent to end-users to tell them the file was sent to the third-party - sequential files are retained, so we can manually do the file transfer again, if required Also, the second mainframe (the third-party) sends back an empty ASCII file, acknowledging receipt of original file. We use this for an audit trail, of sorts. 'tar' was used, so we didn't have to do individual file transfers. It was a vendor requirement to try to keep the files to together. All of this is triggered from EOM or ad-hoc batch jobs. It works well (err, okay so it took a while to get it there!). Hope this helps. Regards David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stuart Boydell Sent: Friday, February 06, 2004 12:01 PM To: U2-Users Subject: [UV/AIX] gzip data to file? [UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped files, the files then being put on a web server for collection. I'm just wondering what the best approach to this might be? How would or have other people approached this? I'm trying to avoid writing the data until after it has been zipped. Rough ideas being... sh - c 'uv prog | gzip -c9 > file' Print to a script? Write to a device (or pipe?)? Output to a java widget? Other? Regards, Stuart Boydell ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
[UV/AIX] gzip data to file?
[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped files, the files then being put on a web server for collection. I'm just wondering what the best approach to this might be? How would or have other people approached this? I'm trying to avoid writing the data until after it has been zipped. Rough ideas being... sh - c 'uv prog | gzip -c9 > file' Print to a script? Write to a device (or pipe?)? Output to a java widget? Other? Regards, Stuart Boydell ** This email message and any files transmitted with it are confidential and intended solely for the use of addressed recipient(s). If you have received this email in error please notify the Spotless IS Support Centre (61 3 9269 7555) immediately who will advise further action. This footnote also confirms that this email message has been scanned for the presence of computer viruses. ** -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
