Re: [UV/AIX] gzip data to file?

2004-02-05 Thread Craig Bennett
>sh - c 'uv prog | gzip -c9 > file'
A disadvantage of this approach is that an run time errors get merged with
your data and how do you know if prog runs to completion?

>Point taken, however, I'm an optimist and not to mention, we paid a giddy
>sum for machines and networks with boat loads of disk space and
impressively
>marketed reliability features ;-)
But writing 10 MB to disk is a problem?


Craig

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Hona, David S
Hehehe! I like that attitude. 

Stuart, next you'll be telling us that you write optimistic code too...who
needs error checking and handling?!?! LOL

Experience makes you wary and somewhat cynical!

regards,
David

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 4:20 PM
To: U2 Users Discussion List
Subject: RE: [UV/AIX] gzip data to file?


> David has a point.  ...  So if the ftp fails, the file still exists.
Point taken, however, I'm an optimist and not to mention, we paid a giddy
sum for machines and networks with boat loads of disk space and impressively
marketed reliability features ;-)

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Stuart Boydell
> David has a point.  ...  So if the ftp fails, the file still exists.
Point taken, however, I'm an optimist and not to mention, we paid a giddy
sum for machines and networks with boat loads of disk space and impressively
marketed reliability features ;-)



**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support Centre (61 3 9269 
7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned
for the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread djordan
Stuart

David has a point.  What we do is a writeseq to a disk file and once
this is completed the UVBasic program executes a unix routine to do an
ftp of the file to another position.  So if the ftp fails, the file
still exists.  The BASIC  Program also deletes the previous file as a
process of self maintenance

Regards

David Jordan

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Hona, David S
Sent: Friday, 6 February 2004 2:32 PM
To: 'U2 Users Discussion List'
Subject: RE: [UV/AIX] gzip data to file?


Stuart,

Ah, one disadvantage, of doing the 'all-in-one' approach is that if one
step fails, you have to start again. A real problem if the output takes
hours to produce and it is part of database update phase too. :-(

For example, if your compression step failed, due to disk space running
out
- will you have to run everything again?

Our sequential file generation can be re-run also, as it doesn't do any
updating itself. This is handy also, in case there is a data issue.

Nothing worse that having to restore data, just to you start again. 

We're experienced about every problem you can think of, hence we're
broken up in to 'repeatable' steps. Works for us.

Regards,
David


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 1:06 PM
To: U2 Users Discussion List
Subject: RE: [UV/AIX] gzip data to file?


> If you can, allow the spooler subsystem to do the work.
Cheers Ray, that seems like an efficient way to do it.

> We create a 80MB sequential file (plus a few small ones) with a UV 
> BASIC
program.
Thanks David, I'm thinking something similar but that outputing to a
device or spooler with a processing script might be the way to go for
us. I'm trying to keep the IO to a minimum if possible. At least if the
spooler is doing any it's at a lower (faster?) level than writeseq.

> why the aversion to writing the data to disk?
I don't think there's a need to write it until the data is in its final
shape. The system already has enough IO to deal with and in theory
should be able to keep the data in memory, which may or may not use disk
caching, but which should be a lighter touch on the system. S

-- 
u2-users mailing list
[EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Hona, David S
Stuart,

Ah, one disadvantage, of doing the 'all-in-one' approach is that if one step
fails, you have to start again. A real problem if the output takes hours to
produce and it is part of database update phase too. :-(

For example, if your compression step failed, due to disk space running out
- will you have to run everything again?

Our sequential file generation can be re-run also, as it doesn't do any
updating itself. This is handy also, in case there is a data issue.

Nothing worse that having to restore data, just to you start again. 

We're experienced about every problem you can think of, hence we're broken
up in to 'repeatable' steps. Works for us.

Regards,
David


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 1:06 PM
To: U2 Users Discussion List
Subject: RE: [UV/AIX] gzip data to file?


> If you can, allow the spooler subsystem to do the work.
Cheers Ray, that seems like an efficient way to do it.

> We create a 80MB sequential file (plus a few small ones) with a UV BASIC
program.
Thanks David, I'm thinking something similar but that outputing to a device
or spooler with a processing script might be the way to go for us. I'm
trying to keep the IO to a minimum if possible. At least if the spooler is
doing any it's at a lower (faster?) level than writeseq.

> why the aversion to writing the data to disk?
I don't think there's a need to write it until the data is in its final
shape. The system already has enough IO to deal with and in theory should be
able to keep the data in memory, which may or may not use disk caching, but
which should be a lighter touch on the system.
S

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Logan, David (SST - Adelaide)
Hi David (sounds familiar doesn't it!)

Perhaps if you were to try zip instead of gzip, as this has a slightly
different methodology, that might cure the issue with the unzip on
windows. You can get it from
http://aixpdslib.seas.ucla.edu/aixpdslib.html

Regards

David Logan
Database Administrator
HP Managed Services
139 Frome Street,
Adelaide 5000
Australia

+61 8 8408 4273
+61 417 268 665



-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of djordan
Sent: Friday, 6 February 2004 1:25 PM
To: 'U2 Users Discussion List'
Subject: RE: [UV/AIX] gzip data to file?


Hi David

Have you found gzip from AIX reliable.  I have one client who would gzip
the universe sequential file (30MB) and then unzip it on a Win NT4
server and we had a high level of corruption.  They don't compress and
they transmitt the files in the evening now.  I don't know if this was
the gzip utility or the Microsoft unzip, although I tried several zip
utitlities on Windows and they all ran into the same problem.

Regards

David Jordan
Dacono Holdings Pty Ltd

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Hona, David S
Sent: Friday, 6 February 2004 12:24 PM
To: 'U2 Users Discussion List'
Subject: RE: [UV/AIX] gzip data to file?


Stuart,

We do something similar, but it involves file transfers from a UNIX
server to a mainframe. Which then transfers the file to yet another
mainframe. Finally, the second mainfame sends it to UNIX server. Don't
ask me why! But there are firewalls, third-party vendors involved, etc.,
etc.

We create a 80MB sequential file (plus a few small ones) with a UV BASIC
program.

Using a shell script:
- copy the above file, plus some others into a staging directory
- the files are individually compressed using standard UNIX compress (to
keep it simple, the third-party didn't or want to use gzip)
- all the files are archived into a standard 'tar' archive utility - to
disk (again, for simplicity)
- the 'tar' file is then transferred to the mainframe via another
external shell script
- email notifications are sent to end-users to tell them the file was
sent to the third-party
- sequential files are retained, so we can manually do the file transfer
again, if required

Also, the second mainframe (the third-party) sends back an empty ASCII
file, acknowledging receipt of original file. We use this for an audit
trail, of sorts.

'tar' was used, so we didn't have to do individual file transfers. It
was a vendor requirement to try to keep the files to together.

All of this is triggered from EOM or ad-hoc batch jobs. It works well
(err, okay so it took a while to get it there!).

Hope this helps.

Regards
David


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 12:01 PM
To: U2-Users
Subject: [UV/AIX] gzip data to file?


[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into
zipped files, the files then being put on a web server for collection.
I'm just wondering what the best approach to this might be? How would or
have other people approached this? I'm trying to avoid writing the data
until after it has been zipped.

Rough ideas being...
sh - c 'uv prog | gzip -c9 > file'
Print to a script?
Write to a device (or pipe?)?
Output to a java widget?
Other?

Regards,
Stuart Boydell


















**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support
Centre (61 3 9269 7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned for
the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
-- 
u2-users mailing list
[EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users
--
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread djordan
Hi David

Have you found gzip from AIX reliable.  I have one client who would gzip
the universe sequential file (30MB) and then unzip it on a Win NT4
server and we had a high level of corruption.  They don't compress and
they transmitt the files in the evening now.  I don't know if this was
the gzip utility or the Microsoft unzip, although I tried several zip
utitlities on Windows and they all ran into the same problem.

Regards

David Jordan
Dacono Holdings Pty Ltd

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Hona, David S
Sent: Friday, 6 February 2004 12:24 PM
To: 'U2 Users Discussion List'
Subject: RE: [UV/AIX] gzip data to file?


Stuart,

We do something similar, but it involves file transfers from a UNIX
server to a mainframe. Which then transfers the file to yet another
mainframe. Finally, the second mainfame sends it to UNIX server. Don't
ask me why! But there are firewalls, third-party vendors involved, etc.,
etc.

We create a 80MB sequential file (plus a few small ones) with a UV BASIC
program.

Using a shell script:
- copy the above file, plus some others into a staging directory
- the files are individually compressed using standard UNIX compress (to
keep it simple, the third-party didn't or want to use gzip)
- all the files are archived into a standard 'tar' archive utility - to
disk (again, for simplicity)
- the 'tar' file is then transferred to the mainframe via another
external shell script
- email notifications are sent to end-users to tell them the file was
sent to the third-party
- sequential files are retained, so we can manually do the file transfer
again, if required

Also, the second mainframe (the third-party) sends back an empty ASCII
file, acknowledging receipt of original file. We use this for an audit
trail, of sorts.

'tar' was used, so we didn't have to do individual file transfers. It
was a vendor requirement to try to keep the files to together.

All of this is triggered from EOM or ad-hoc batch jobs. It works well
(err, okay so it took a while to get it there!).

Hope this helps.

Regards
David


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 12:01 PM
To: U2-Users
Subject: [UV/AIX] gzip data to file?


[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into
zipped files, the files then being put on a web server for collection.
I'm just wondering what the best approach to this might be? How would or
have other people approached this? I'm trying to avoid writing the data
until after it has been zipped.

Rough ideas being...
sh - c 'uv prog | gzip -c9 > file'
Print to a script?
Write to a device (or pipe?)?
Output to a java widget?
Other?

Regards,
Stuart Boydell


















**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support
Centre (61 3 9269 7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned for
the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
-- 
u2-users mailing list
[EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Stuart Boydell
> If you can, allow the spooler subsystem to do the work.
Cheers Ray, that seems like an efficient way to do it.

> We create a 80MB sequential file (plus a few small ones) with a UV BASIC
program.
Thanks David, I'm thinking something similar but that outputing to a device
or spooler with a processing script might be the way to go for us. I'm
trying to keep the IO to a minimum if possible. At least if the spooler is
doing any it's at a lower (faster?) level than writeseq.

> why the aversion to writing the data to disk?
I don't think there's a need to write it until the data is in its final
shape. The system already has enough IO to deal with and in theory should be
able to keep the data in memory, which may or may not use disk caching, but
which should be a lighter touch on the system.
S



**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support Centre (61 3 9269 
7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned
for the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


Re: [UV/AIX] gzip data to file?

2004-02-05 Thread Craig Bennett
>[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into
zipped
>files, the files then being put on a web server for collection. I'm just
>wondering what the best approach to this might be? How would or have other
>people approached this? I'm trying to avoid writing the data until after it
>has been zipped.

Stuart,

why the aversion to writing the data to disk?

Write the data to a type 19 file and then for each file execute the shell
command "gzip FILE"
This will cause gzip replace the original file with the compressed data
FILE.gz.

I think your shell example will work (and it would be my preffered option
for thos you suggest) but my suggestion may be simpler to maintain.


Craig

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


Re: [UV/AIX] gzip data to file?

2004-02-05 Thread Ray Daignault
If you can, allow the spooler subsystem to do the work.

Spool the data, and write a script for the spooler to execute.  Compress the
data and ftp where you wish it to go.

Remember that the spooler runs as root, so you may have some NFS issues
should you attempt to do a "cp" to an nfs mounted partition.

Regards,

Ray D
- Original Message - 
From: "Stuart Boydell" <[EMAIL PROTECTED]>
To: "U2-Users" <[EMAIL PROTECTED]>
Sent: Thursday, February 05, 2004 5:00 PM
Subject: [UV/AIX] gzip data to file?


> [UV10/AIX] We have a requirement to output (10MB +/- ascii) data into
zipped
> files, the files then being put on a web server for collection. I'm just
> wondering what the best approach to this might be? How would or have other
> people approached this? I'm trying to avoid writing the data until after
it
> has been zipped.
>
> Rough ideas being...
> sh - c 'uv prog | gzip -c9 > file'
> Print to a script?
> Write to a device (or pipe?)?
> Output to a java widget?
> Other?
>
> Regards,
> Stuart Boydell
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> **
> This email message and any files transmitted with it are confidential
> and intended solely for the use of addressed recipient(s). If you have
> received this email in error please notify the Spotless IS Support Centre
(61 3 9269 7555) immediately who will advise further action.
>
> This footnote also confirms that this email message has been scanned
> for the presence of computer viruses.
> **
>
> -- 
> u2-users mailing list
> [EMAIL PROTECTED]
> http://www.oliver.com/mailman/listinfo/u2-users
>

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


RE: [UV/AIX] gzip data to file?

2004-02-05 Thread Hona, David S
Stuart,

We do something similar, but it involves file transfers from a UNIX server
to a mainframe. Which then transfers the file to yet another mainframe.
Finally, the second mainfame sends it to UNIX server. Don't ask me why! But
there are firewalls, third-party vendors involved, etc., etc.

We create a 80MB sequential file (plus a few small ones) with a UV BASIC
program.

Using a shell script:
- copy the above file, plus some others into a staging directory
- the files are individually compressed using standard UNIX compress (to
keep it simple, the third-party didn't or want to use gzip)
- all the files are archived into a standard 'tar' archive utility - to disk
(again, for simplicity)
- the 'tar' file is then transferred to the mainframe via another external
shell script
- email notifications are sent to end-users to tell them the file was sent
to the third-party
- sequential files are retained, so we can manually do the file transfer
again, if required

Also, the second mainframe (the third-party) sends back an empty ASCII file,
acknowledging receipt of original file. We use this for an audit trail, of
sorts.

'tar' was used, so we didn't have to do individual file transfers. It was a
vendor requirement to try to keep the files to together.

All of this is triggered from EOM or ad-hoc batch jobs. It works well (err,
okay so it took a while to get it there!).

Hope this helps.

Regards
David


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Stuart Boydell
Sent: Friday, February 06, 2004 12:01 PM
To: U2-Users
Subject: [UV/AIX] gzip data to file?


[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped
files, the files then being put on a web server for collection. I'm just
wondering what the best approach to this might be? How would or have other
people approached this? I'm trying to avoid writing the data until after it
has been zipped.

Rough ideas being...
sh - c 'uv prog | gzip -c9 > file'
Print to a script?
Write to a device (or pipe?)?
Output to a java widget?
Other?

Regards,
Stuart Boydell


















**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support Centre
(61 3 9269 7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned
for the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users
-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users


[UV/AIX] gzip data to file?

2004-02-05 Thread Stuart Boydell
[UV10/AIX] We have a requirement to output (10MB +/- ascii) data into zipped
files, the files then being put on a web server for collection. I'm just
wondering what the best approach to this might be? How would or have other
people approached this? I'm trying to avoid writing the data until after it
has been zipped.

Rough ideas being...
sh - c 'uv prog | gzip -c9 > file'
Print to a script?
Write to a device (or pipe?)?
Output to a java widget?
Other?

Regards,
Stuart Boydell


















**
This email message and any files transmitted with it are confidential
and intended solely for the use of addressed recipient(s). If you have 
received this email in error please notify the Spotless IS Support Centre (61 3 9269 
7555) immediately who will advise further action.

This footnote also confirms that this email message has been scanned
for the presence of computer viruses.
**

-- 
u2-users mailing list
[EMAIL PROTECTED]
http://www.oliver.com/mailman/listinfo/u2-users