Firstly, thanks for everyone's input and suggestions on this wish list item
I've had for years.
Nextly, I jumped into VMFTP because of the great comments, and also because
there was an entire attached program which I was able to get to my CMS userid
and after tweaks and local site fiddling, had it working.
I got so consumed with getting it to work that I didn't realize until I was
done that there doesn't not appear to be any way to divert the data into a
pipeline, so all I had in the end was an alternative method to a straight FTP
which I wasn't having any problem to begin with. Also, the Kbyte/sec transfer
rate was consistently less that the FTP MODULE off of our MAINT disk.
So, I turned back to the first suggestion, Steve Hayes' FTP stage that is
flagged as "New" on the VM download site.
Took about a week (part time as much as possible) to get it working correctly.
Our company has an MVS userid convention of the first character of userids
begin with @.
Needless to say, that silly @ part of the MVS userid caused a number of
unwanted results during the URL processing of the arguments to FTP_REXX.
But it is now working, transfer rate is comparable to MAINT's FTP MODULE, and
it works wonderfully as a PIPELINE driver (I've really been just hammering away
with its use as a first stage driver, to read huge MVS datasets, which can even
reside on tape but work nonetheless.
I am one happy camper (thanks for the tip, John H)!!
One thing I've done which seems basically a "must have" for this rexx stage but
I had to add the code to make it work is to detect a severed output and quit
the FTP appropriately/cleanly.
Often times I want just the first 100 - 10,000 records of a dataset, to study
and understand the new record structure, before moving on to processing the
entire dataset, which may even turn into another daily or weekly job that
processes the entire dataset each time. I used to use FILEAID with COPY=1000
to get the first portion of a dataset, creating a temporary small dataset to
have to remember to delete later, and then FTP GET it to VM.
But now I can "pipe ftp_rexx < really.huge.dataset | take 1000 | > partial file
a"
This is REALLY great!! But, the rexx stage was reading the entire dataset,
even though the pipe did output only a 1000 records.
I have gotten around this problem with the following code, and wondered if
anyone has an idea for a better way.
I couldn't think of a way to use PIPCMD, so my solution has two copies of
FTP_REXX running, the 2nd copy just to detect the severed output for the main
pipe in the first.
My additions to the pipe include extra ******* in the comments:
parse source . . ftp . /*
existing line of code, variable "ftp" = name of this rexx stage */
if Arg(1) = 'DETECT0' Then Call Detect_NoMore_Output /******* If
rexx stage called with special argument then go do special processing */
...
...
/* Here's the main pipe of FTP_REXX */
'CALLPIPE (name' ftp || ':' || command 'endchar \)' ,
'i: faninany' , /* pipestoppers here */
'| g: gate' , /* first one terminates */
'\ *.in.data:' , /* data from server */
'| g:' , /* stop at EoF or async cmd */
'| m: fanout' , /* 2nd copy for count */
Deblocker() , /* Post-processor */
xlata2e , /* tranlate ASCII-EBCDIC? */
'| S:' ftp 'DETECT0' , /****** S: label is for a special
secondary output, "ftp" is the name of this entire rexx stage */
'| *.out.0:' , /* output data */
tabla2e , /* translate table input? */
'\ m:' , /* output data */
'| count bytes' , /* waits till EoF */
'| var n' , /* save byte count */
'| i:' , /* close gate to stop pipe */
'\ S:' , /****** My special secondary output
that will get */
'| i:' , /****** fed into the "gate" stage to
close/stop the pipe */
copies('\ m:' , /* postprocessed output copy */
'| addrdw cms4' , /* bytes in this record */
'| spec 1-4 c2d 1' , /* convert to decimal */
'| literal 0' , /* start the counter */
'| spec a: w1 . print #0+=a 1 "bytes received" nw', /* counter */
'| j: juxtapose' , /* write each t seconds */
'| cons' , /* only if noisy is on */
'\ literal +'max(1, timeout%3) , /* counters 3/timeout */
'| dup *' , /* could run for ever */
'| delay' , /* wait for it */
'| chop 0' , /* trigger juxtapose count */
'| g:' , /* */
'| j:' , echo?) ; /* inject into cmd stream */
/*--------------------------------------------------------------------*/
/* Detect_NoMore_Output: If no more outputting, possibly from a */
/* TAKE nnn stage downstream or something, then Sever input and end.*/
/*--------------------------------------------------------------------*/
Detect_NoMore_Output:
Do Forever /******* Read entire
input, for now.*/
'CALLPIPE *: | TAKE 10000 | *:' /******* Read and pass along kind of
big chunks */
'PEEKTO RECORD' /******* Ensure still more input
*/
If rc ¬= 0 Then Leave /******* Get out if not.
*/
'STREAMSTATE OUTPUT 0' /******* Now the big test, see if
primary output stream is still connected */
If WORDPOS(rc,'-4 12') > 0 Then Do /******* and if not,
*/
'SEVER INPUT' /******* Sever input, and pass the
following literal to the secondary output */
'CALLPIPE LITERAL Output Disconnected| *.OUTPUT.1:' /* which
ultimately gets fed into the GATE stage above and stops the pipe */
Leave /******* No more
reading forever... */
End
End
Exit
-----Original Message-----
From: CMSTSO Pipelines Discussion List [mailto:[email protected]] On
Behalf Of John P. Hartmann
Sent: Thursday, August 25, 2011 4:28 PM
To: [email protected]
Subject: Re: FTP user exit, ultimately a PIPE stage?
There is Steve Hayes ftp driver at the pipelines home page. And for
non-pipe solution, see Romney's RXFTP at the VM download site.
On 25 August 2011 18:10, Larson, John E. <[email protected]> wrote:
> I am getting lost in Google-land, and seem to keep hitting dead ends.
>
> Something that I've wanted to do for a really long time but just assumed
> wasn't possible.
>
> Is it even possible...to write an FTP user exit that interfaces with the
> entire retrieval mechanism?
>
> Ultimately, I'd wish to have a PIPE stage that can filter my FTP GET of a
> 3000+ cylinder MVS dataset to a VM disk.
> I've always wondered why I have to find a VM disk large enough to complete
> the FTP GET, so that I can then turn around and "PIPE < FTP FILE | many
> filters | > much less data..."
>
> What would be the first step in even trying to attempt this? An FTP user
> exit? A socket program that does the LOGIN and entire FTP itself (rewriting
> FTP)??
>
> Or is this just too complex to consider?
>
> The larger the MVS dataset (multi-volume), the more awkward current methods
> are, i.e., I use FILEAID to chop the MVS dataset into 5-10 more manageable
> datasets (each less than 3000 cylinders).
> Then I can FTP GET each one, do my PIPE processing, and when all done write a
> final PIPE to merge all of the individual files and combine into one final
> output file.
>
> I suppose that preprocessing on MVS is an obvious alternative, but I wish
> that I could figure out how to do this while staying at home (on VM:-)
>
> John
>
