Thank you so much!!!
On Mon, Jul 8, 2013 at 9:50 AM, Larson, John E. <[email protected]> wrote: > Re-reading my own original response I find that ... I really shouldn't > respond to emails when I've just gotten up and haven't had my coffee yet, > the number of grammatical errors is almost ... embarrassing:-) > > Anyway... > > I find that I made two changes to the original source, which was in all a > total of less than 20 lines of rexx code added/changed. > > 1) Probably not an issue for you at all, but our company has a standard of > naming everyone's MVS userids beginning with an '@'. > For example (close enough to illustrate the problem) my MVS userid is > @JOHN. > Right around line #830 of FTP_REXX is some code that is interpreting the > URL possibly passed to it: > > url: procedure expose (variables) > parse var url scheme '//' local > if local = '' then parse value 'FTP:' url with scheme local > if translate(scheme) ¬= 'FTP:' then call BadURL "invalid URL scheme > '"scheme"'" > parse var local login '/' file > if pos('@', login) = 0 > then parse value 'anonymous' login with login server > else parse var login login '@' server > if server = '' then call BadURL "no host specified" > > That's the area of the code I had to put some extra "stuff" to deal with > our MVS userids beginning with an '@'. > > > 2) Halting after detecting output stream is severed, possibly from a "| > TAKE nnn |" stage or whatever. > > I find that I received a suggestion about this over a year ago, including > this from Mr. Hartmann: > > "EOFREPORT ALL should make your FTP stage unwind properly without any > hokypokery. > The socket is shut down for read when FTPCLIENT sees EOF on its output." > > But alas, I struggled and struggled and couldn't get the EOFREPORT ALL to > work for me, so I had to resort to hokypokery:-) > Seriously...not very glamorous at all, but this has worked fine for me and > I'm not so concerned with the elegant solutions as my work focus is simply > more driven by "getting the job done whatever it takes": > > Original pipe rexx code: > > 02118 if ok? /* We are happy, so > get the */ > 02119 then do /* data which is > queuing up */ > 02120 call xlatestages (type = 'A') /* translate for ASCII > */ > 02121 call time('R') /* in the data stream > */ > 02122 'CALLPIPE (name' ftp':'command 'end \ listerr)', > 02123 'i: faninany' , /* pipestoppers here > */ > 02124 '| g: gate strict' , /* first one terminates > */ > 02125 '\ *.in.data:' , /* data from server > */ > 02126 '| g:' , /* stop at EoF > or async cmd */ > 02127 '| m: fanout' , /* 2nd copy for count > */ > 02128 Deblocker() , /* Post-processor > */ > 02129 xlata2e , /* tranlate > ASCII-EBCDIC? */ > 02130 '| *.out.0:' , /* output data > */ > 02131 tabla2e , /* translate table > input? */ > 02132 '\ m:' , /* output data > */ > 02133 '| count bytes' , /* waits till EoF > */ > 02134 '| var n' , /* save byte count > */ > 02135 '| i:' , /* close gate > to stop pipe */ > 02136 copies('\ m:' , /* postprocessed output > copy */ > ... > ... > > And here's my hokypokery: > > 'CALLPIPE (name' ftp':'command 'end \ listerr)' , > 'i: faninany' , /* pipestoppers here */ > '| g: gate' , /* first one terminates > */ > '\ *.in.data:' , /* data from server */ > '| g:' , /* stop at EoF or async > cmd */ > '| m: fanout' , /* 2nd copy for count */ > Deblocker() , /* Post-processor */ > xlata2e , /* tranlate ASCII-EBCDIC? > */ > '| s:' ftp 'DETECT0' , /* John - How did I even come up with > this?? */ > '| *.out.0:' , /* output data > */ > tabla2e , /* translate table input? > */ > '\ m:' , /* output data > */ > '| count bytes' , /* waits till EoF */ > '| var n' , /* save byte count > */ > '| i:' , /* close gate to stop > pipe */ > '\ s:' , /* John (fanout of sorts??) output to detect no more > output */ > '| i:' , /* close gate to stop > pipe */ > copies('\ m:' , /* postprocessed output copy */ > ... > ... > > At the very top of the entire Rexx stage/program: > > if Arg(1) = 'DETECT0' Then Call DETECT_NOMORE_OUTPUT > > A hokie (but heck, I'm an Okie from Oklahoma) sub-routine at the end of > the program: > > /*--------------------------------------------------------------------*/ > /* Detect_NoMore_Output: If no more outputting, possibly from a */ > /* TAKE nnn stage downstream or something, then Sever input and end.*/ > /*--------------------------------------------------------------------*/ > DETECT_NOMORE_OUTPUT: > Do Forever > 'CALLPIPE *: | TAKE 10000 | *:' , /* An arbitrary choice to > pause briefly */ > 'PEEKTO RECORD' , /* after 10K > records and use a PEEKTO */ > If rc ¬= 0 Then Leave > 'STREAMSTATE OUTPUT 0' > If WORDPOS(rc,'-4 12') > 0 Then Do > 'SEVER INPUT' > 'CALLPIPE LITERAL Output Disconnected| *.OUTPUT.1:' > Leave > End > End > Exit > > Other than those two changes, made a year and a half ago, I have used > FTP_REXX thousands of times now, reading and filtering/manipulating > countless hundreds of millions of MVS records by now, with no other > modifications since. > > Keep in mind that I had no interest in trying to read any pages/data from > external (to our company) sites. > > I wanted very specifically a solution to the way I was FTPing HUGE MVS > datasets to VM, waiting for the entire file to be transferred, before I > could even begin to start using built-in stages or writing Rexx/pipe stages > to process the data. > > For my purpose, I have been a happy camper with FTP_REXX, which definitely > makes my life easier... > > John > > > -----Original Message----- > From: CMSTSO Pipelines Discussion List [mailto:[email protected]] > On Behalf Of Hobart Spitz > Sent: Sunday, July 07, 2013 9:18 AM > To: [email protected] > Subject: Re: FTP REXX > > John; > > Thanks. This is great info. Is there any chance you could upload or post > the change for a disconnected output stream (and anything else you think is > useful)? > > Thanks. > > - Hobart > > > On Sun, Jul 7, 2013 at 10:24 AM, Larson, John E. <[email protected]> > wrote: > > > I put out a similar query about a year ago, and was told that this was > > very workable. > > > > I took me 1-2 weeks (difficult to find time) to make some minor > > internal changes that were required to make it work in our specific > > environment with our unique firewalls and such. > > > > But now that I have it working I love it and wanted this for a long time. > > > > No longer to I have to transfer 10 - 100 million record files from MVS > > to VM (using a full pack mod 9 available to me) so that I can then run > > my pipe filter against the file to extract the thousand or whatever > records. > > > > Simply 'PIPE FTP host mvs_dsn | filter | > final file b' > > > > And a real beauty is I don't have to bother with FILEAID or other jobs > > if I just want the first thousand records for analysis, simply 'PIPE > > FTP host mvs_dsn '| TAKE 1000 | ... > > However, I seem to recall that I had to trace the FTP rexx code and > > make a change to make it exit when it detected the output stream > > wasn't connected any longer ... pretty sure I had to do this... > > > > And yes, I use it in a multi-stream pipe and have one instance pulling > > a DSN from one MVS system and another instance running simultaneously > > pulling a DSN from a separate MVS system, merging and processing the > > two DSNs as one, with filters, I was SO HAPPY when this was suddenly > working. > > > > I do so much MVS --> VM FTP work in the course of a work day that I > > wished for an FTP rexx stage for some years, finally dove into the one > > I'm sure you're referring to and have been enjoying it ever since. > > > > John > > > > > > -----Original Message----- > > From: CMSTSO Pipelines Discussion List > > [mailto:[email protected]] > > On Behalf Of Hobart Spitz > > Sent: Sunday, July 07, 2013 4:46 AM > > To: [email protected] > > Subject: FTP REXX > > > > Does anyone know the status of FTP REXX? It's marked as new on the > > Pipes web page, but the doc and change history dates are more than 10 > years old. > > > > Is anyone using it? Does anyone know if it still works? > > > > Can FTP REXX run under PipeServ? Can multiple instances run > > successfully at the same time? > > > > I'd be happy to get educated guesses in the absence of firm answers. > > > > Thanks all. > > > > -- > > OREXXMan > > > > > > -- > OREXXMan > -- OREXXMan
