Re-reading my own original response I find that ... I really shouldn't respond
to emails when I've just gotten up and haven't had my coffee yet, the number of
grammatical errors is almost ... embarrassing:-)
Anyway...
I find that I made two changes to the original source, which was in all a total
of less than 20 lines of rexx code added/changed.
1) Probably not an issue for you at all, but our company has a standard of
naming everyone's MVS userids beginning with an '@'.
For example (close enough to illustrate the problem) my MVS userid is @JOHN.
Right around line #830 of FTP_REXX is some code that is interpreting the URL
possibly passed to it:
url: procedure expose (variables)
parse var url scheme '//' local
if local = '' then parse value 'FTP:' url with scheme local
if translate(scheme) ¬= 'FTP:' then call BadURL "invalid URL scheme
'"scheme"'"
parse var local login '/' file
if pos('@', login) = 0
then parse value 'anonymous' login with login server
else parse var login login '@' server
if server = '' then call BadURL "no host specified"
That's the area of the code I had to put some extra "stuff" to deal with our
MVS userids beginning with an '@'.
2) Halting after detecting output stream is severed, possibly from a "| TAKE
nnn |" stage or whatever.
I find that I received a suggestion about this over a year ago, including this
from Mr. Hartmann:
"EOFREPORT ALL should make your FTP stage unwind properly without any
hokypokery.
The socket is shut down for read when FTPCLIENT sees EOF on its output."
But alas, I struggled and struggled and couldn't get the EOFREPORT ALL to work
for me, so I had to resort to hokypokery:-)
Seriously...not very glamorous at all, but this has worked fine for me and I'm
not so concerned with the elegant solutions as my work focus is simply more
driven by "getting the job done whatever it takes":
Original pipe rexx code:
02118 if ok? /* We are happy, so get
the */
02119 then do /* data which is queuing up
*/
02120 call xlatestages (type = 'A') /* translate for ASCII
*/
02121 call time('R') /* in the data stream
*/
02122 'CALLPIPE (name' ftp':'command 'end \ listerr)',
02123 'i: faninany' , /* pipestoppers here
*/
02124 '| g: gate strict' , /* first one terminates */
02125 '\ *.in.data:' , /* data from server
*/
02126 '| g:' , /* stop at EoF or
async cmd */
02127 '| m: fanout' , /* 2nd copy for count */
02128 Deblocker() , /* Post-processor */
02129 xlata2e , /* tranlate ASCII-EBCDIC?
*/
02130 '| *.out.0:' , /* output data
*/
02131 tabla2e , /* translate table input?
*/
02132 '\ m:' , /* output data
*/
02133 '| count bytes' , /* waits till EoF */
02134 '| var n' , /* save byte count
*/
02135 '| i:' , /* close gate to
stop pipe */
02136 copies('\ m:' , /* postprocessed output copy
*/
...
...
And here's my hokypokery:
'CALLPIPE (name' ftp':'command 'end \ listerr)' ,
'i: faninany' , /* pipestoppers here */
'| g: gate' , /* first one terminates */
'\ *.in.data:' , /* data from server */
'| g:' , /* stop at EoF or async cmd
*/
'| m: fanout' , /* 2nd copy for count */
Deblocker() , /* Post-processor */
xlata2e , /* tranlate ASCII-EBCDIC? */
'| s:' ftp 'DETECT0' , /* John - How did I even come up with this?? */
'| *.out.0:' , /* output data */
tabla2e , /* translate table input? */
'\ m:' , /* output data
*/
'| count bytes' , /* waits till EoF */
'| var n' , /* save byte count */
'| i:' , /* close gate to stop pipe
*/
'\ s:' , /* John (fanout of sorts??) output to detect no more
output */
'| i:' , /* close gate to stop pipe
*/
copies('\ m:' , /* postprocessed output copy */
...
...
At the very top of the entire Rexx stage/program:
if Arg(1) = 'DETECT0' Then Call DETECT_NOMORE_OUTPUT
A hokie (but heck, I'm an Okie from Oklahoma) sub-routine at the end of the
program:
/*--------------------------------------------------------------------*/
/* Detect_NoMore_Output: If no more outputting, possibly from a */
/* TAKE nnn stage downstream or something, then Sever input and end.*/
/*--------------------------------------------------------------------*/
DETECT_NOMORE_OUTPUT:
Do Forever
'CALLPIPE *: | TAKE 10000 | *:' , /* An arbitrary choice to pause
briefly */
'PEEKTO RECORD' , /* after 10K records
and use a PEEKTO */
If rc ¬= 0 Then Leave
'STREAMSTATE OUTPUT 0'
If WORDPOS(rc,'-4 12') > 0 Then Do
'SEVER INPUT'
'CALLPIPE LITERAL Output Disconnected| *.OUTPUT.1:'
Leave
End
End
Exit
Other than those two changes, made a year and a half ago, I have used FTP_REXX
thousands of times now, reading and filtering/manipulating countless hundreds
of millions of MVS records by now, with no other modifications since.
Keep in mind that I had no interest in trying to read any pages/data from
external (to our company) sites.
I wanted very specifically a solution to the way I was FTPing HUGE MVS datasets
to VM, waiting for the entire file to be transferred, before I could even begin
to start using built-in stages or writing Rexx/pipe stages to process the data.
For my purpose, I have been a happy camper with FTP_REXX, which definitely
makes my life easier...
John
-----Original Message-----
From: CMSTSO Pipelines Discussion List [mailto:[email protected]] On
Behalf Of Hobart Spitz
Sent: Sunday, July 07, 2013 9:18 AM
To: [email protected]
Subject: Re: FTP REXX
John;
Thanks. This is great info. Is there any chance you could upload or post the
change for a disconnected output stream (and anything else you think is useful)?
Thanks.
- Hobart
On Sun, Jul 7, 2013 at 10:24 AM, Larson, John E. <[email protected]> wrote:
> I put out a similar query about a year ago, and was told that this was
> very workable.
>
> I took me 1-2 weeks (difficult to find time) to make some minor
> internal changes that were required to make it work in our specific
> environment with our unique firewalls and such.
>
> But now that I have it working I love it and wanted this for a long time.
>
> No longer to I have to transfer 10 - 100 million record files from MVS
> to VM (using a full pack mod 9 available to me) so that I can then run
> my pipe filter against the file to extract the thousand or whatever records.
>
> Simply 'PIPE FTP host mvs_dsn | filter | > final file b'
>
> And a real beauty is I don't have to bother with FILEAID or other jobs
> if I just want the first thousand records for analysis, simply 'PIPE
> FTP host mvs_dsn '| TAKE 1000 | ...
> However, I seem to recall that I had to trace the FTP rexx code and
> make a change to make it exit when it detected the output stream
> wasn't connected any longer ... pretty sure I had to do this...
>
> And yes, I use it in a multi-stream pipe and have one instance pulling
> a DSN from one MVS system and another instance running simultaneously
> pulling a DSN from a separate MVS system, merging and processing the
> two DSNs as one, with filters, I was SO HAPPY when this was suddenly working.
>
> I do so much MVS --> VM FTP work in the course of a work day that I
> wished for an FTP rexx stage for some years, finally dove into the one
> I'm sure you're referring to and have been enjoying it ever since.
>
> John
>
>
> -----Original Message-----
> From: CMSTSO Pipelines Discussion List
> [mailto:[email protected]]
> On Behalf Of Hobart Spitz
> Sent: Sunday, July 07, 2013 4:46 AM
> To: [email protected]
> Subject: FTP REXX
>
> Does anyone know the status of FTP REXX? It's marked as new on the
> Pipes web page, but the doc and change history dates are more than 10 years
> old.
>
> Is anyone using it? Does anyone know if it still works?
>
> Can FTP REXX run under PipeServ? Can multiple instances run
> successfully at the same time?
>
> I'd be happy to get educated guesses in the absence of firm answers.
>
> Thanks all.
>
> --
> OREXXMan
>
--
OREXXMan
