An actual problem with FTP transfers losing data can occur when you have the receiving system polling for the target file's existance; if the poll finds the file exists, and starts to use it (especially to copy it) before the FTP completes, you may wind up with
- a partial file - a full length file with partial data - a zero-length file All depends on OS, ftp implementation, etc. This is based on actual experiences with FTPing large files from slow systems, with much faster recipient systems. Dave A. -----Original Message----- From: Adiraju, Rao [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 3:50 PM To: [EMAIL PROTECTED] Subject: Re: A novice question--THE SAGA CONTINUES Ray & Rick Even I haven't heard FTP loosing the data. As I said earlier, we opted for a PULL rather than PUSH. Because for a PUSH every client (or server for that matter) have to have a main-frame user-id and passwords, which was a BIG NO NO at my previous client for security reasons. They were more paranoid (obviously so) exposing our main-frames than others. We also had situations where some errors from FTP were not getting translated in to error return codes on to batch JCL steps but with some JES2 commands, that was tightened with FTP error handling. So for example any of the FTP commands fail (OPEN, userid, password, filenotfound etc..) it was setting the job step error codes appropriately. Alternatively use MQ with blocking de-blocking. Have fun. Rao -----Original Message----- From: Rick Tsujimoto [mailto:[EMAIL PROTECTED] Sent: 4 March 2004 7:18 AM To: [EMAIL PROTECTED] Subject: Re: A novice question--THE SAGA CONTINUES Raymond, FTP may not "lose" data, but it sure can return a false positive. For example, if your FTP password is wrong, FTP will still return a good return code even though the FTP operation never completed. This situation was the reason a client of mine had me develop an FTP-like facility under MQ. It's not the best use of MQ, but certainly do-able. There are lots of MQ shops that do batch-like processing in addition to near real-time processing (let's not go down that road again...). My personal take on this is to stick with FTP when the processing is truly batch, but you may have to develop a wrapper around FTP to make doubly sure it did work. Or, develop a combination of the two - I think someone mentioned reference messages. As for speed, I spent a lot of time tweaking my MQ-FTP like solution under MQ V1.1.4 and came to the conclusion that the only way to gain significant speed was to use non-persistent messages and to develop code to block and deblock messages. The speed was almost as good as FTP. "Kinzler, Raymond C" To: [EMAIL PROTECTED] <RaymondCKinzler cc: @EATON.COM> Subject: Re: A novice question--THE SAGA CONTINUES Sent by: MQSeries List <[EMAIL PROTECTED] en.AC.AT> 03/03/2004 08:30 AM Please respond to MQSeries List T.Rob--I agree with you. It's trying to convince a single person here that FTP is a reliable data transfer method. It is also a chore to get this person to understand the difference between a real-time application vs. a batch application. Maybe the REAL problem is that this person has been a programmer for years and years and has recently made the switch to business analyst and has subsequently forgotten everything that was learned! I don't think my proposal is the sexiest one that has ever been thought of but it is a solid and reliable solution which is better than what we have currently. It is a long, drawn-out story which won't add any value to this group. What you wrote WILL help me in my negotiations because it adds a sort of second opinion that is valued by this person. Thanks. -----Original Message----- From: Wyatt, T. Rob [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 8:06 AM To: [EMAIL PROTECTED] Subject: Re: A novice question--THE SAGA CONTINUES Raymond, Don't know if this will help you with your negotiations but FTP clients typically have error correction built in. It would be a very old (or very bad) implementation of FTP if, at the very least, it did not compare checksums after the transfer and provide a completion status. The newest versions include error checking on each packet, automatic retry and restart, resume from interrupted transfer and, in the case of SFTP, SSL which provides encryption, authentication and file integrity. The FTP you are using would quite literally have to be 80-ish (or even earlier) to suffer from the problems that have been attributed to it. -- T.Rob -----Original Message----- From: Kinzler, Raymond C [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 03, 2004 7:39 AM To: [EMAIL PROTECTED] Subject: Re: A novice question--THE SAGA CONTINUES Rao, Basically, this is what I was saying to do. My solution was a tad different in that I proposed to use something that already exists whereby an Excel file is translated into a .TXT file and the .TXT file is FTP'd to the mainframe and a GDG is created on the mainframe. When the file is catalogued, a job gets kicked off that processes the file and updates the database. A report is created along the way that is e-mailed to the user who initiated the process. The so-called 'problem' with doing it this way was (a) the application I speak of is a java package that processes about 15 or so spreadsheets and it's a pain to keep everybody that uses it up-to-date and (b) people are saying that FTP may 'lose' records while MQ guarantees 100% delivery. My counterpoint to the first 'problem' is to get rid of that java package that all the users load onto their PC and use the server that they are using to transmit the messages via MQ. There is no reason the application they use cannot simply FTP a file instead of using MQ. That would eliminate the problem of keeping individual machines up-to-date (which is virtually impossible because the machines we are talking about are outside users in may cases and 100% of the time for this specific application). As for the second 'problem,' I do not agree with the assumption that FTP 'loses' records as we DBAs run a set of processes every night that easily sends hundreds-of-thousands (and sometimes over a million) records to our data warehouse and we have never experienced any sort of data loss. Like you said, this solution is so, so, 80's-ish and people seem to want spiffy, new, flashy solutions. I am more of the type of person who wants to see a solid solution, which your suggestion is. Thanks a lot! I have gotten more out of discussions on this group than all the pages of the MQ manuals I have read. *** Credit Lyonnais **************************************************** This e-mail contains confidential information or information belonging to Credit Lyonnais and is intended solely for the addressees. The unauthorized disclosure, use, dissemination or copying (either whole or partial) of this e-mail, or any information it contains, is prohibited. E-mails are susceptible to alteration and their integrity cannot be guaranteed. Credit Lyonnais shall not be liable for this e-mail if modified or falsified. If you are not the intended recipient of this e-mail, please delete it immediately from your system and notify the sender of the wrong delivery and the mail deletion. Credit Lyonnais in the Americas: Credit Lyonnais Bank New York Branch, Credit Lyonnais Americas Services Inc., Credit Lyonnais Rouse (USA) Ltd., and Credit Lyonnais Securities (USA) Inc. *** Credit Lyonnais **************************************************** Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com Archive: http://vm.akh-wien.ac.at/MQSeries.archive
