Re: [twsocket] FTP Client - reconnecting
On 07/11/2013 16:15, Angus Robertson - Magenta Systems Ltd wrote: Angus, trying to change the code to use your components. Does the TMagFtp component support a CopyType := FCTypeDates ; Sorry, seems like I never supported specific date ranges for FTP which is mostly used to synchronise remote and local directories, or for specific files. The following code inserted in to TMagFTP.FtpDownFiles seems to do the job I need if (CopyType = FCTypeDates) and (FEarliestDate > 0) then begin for i := 0 to Pred(TotSrcFiles) do begin SrcFileRec := SrcFileList[i]; with SrcFileRec^ do begin if FrFileCopy = FCStateSelect then begin if FrFileUDT < FEarliestDate then begin FrFileCopy := FCStateNone; Dec(fTotProcFiles); end; end; end; end; if fTotProcFiles = 0 then begin result := TaskResOKNone; fReqResponse := 'No Specific Source Files Selected to Download'; Exit; end; end; David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
Angus Turned everything firewall like off and it makes no difference, and I assume that wouldn't explain why when downloading the same root folder each time, it would fail in a different place (sometimes not at all). I'm pretty sure it's something server side, but it's hosted by big US ISP, so not much chance of getting logs. I'll contact their support though and see if I can get anything. Thanks David On 07/11/2013 18:02, Angus Robertson - Magenta Systems Ltd wrote: Or maybe the FTP server has a badly implemented session timeout? Can you get the server logs or is not your server? -- David Rose Navigator Systems Ltd, Network Business Centre, 9 River Street Place, Julian Road, Bath BA1 2RS Main Tel: +44 (0)207 183 0011 Fax: +44 (0)1225 464984 Company Email: da...@navigator.co.uk (Private e-mail on request) CONFIDENTIALITY NOTICE The contents of this e-mail are confidential to the ordinary user of the e-mail address to which it was addressed and may also be privileged. If you are not the addressee of this e-mail you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this e-mail in error please e-mail the sender by replying to this message. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
> *Subject:* Re: [twsocket] FTP Client - reconnecting > Now trying with your components and although it's much better > behaved and fails gracefully, it still fails (in my app or the > supplied demo app). > However as both your app and my app fail with my hosted FTP server > and both work with your FTP server, I'm going to assume that > there's something amiss with the ISP's FTP offering. Any clues in > the log as to what it could be ? > > FTP Session Closed > Downloaded File c:\ftp_test\hiretrack nx\3.0.1.151\hiretrack > nx_cab.tmp, Size 321,058 > Request Failed: Partial File Downloaded, Resume Allowed, Expected > File Size 12,325,384, Actual Size 321,058 Looks like the session closed after two seconds of downloading, once 300K of data had been done. No reason for the server or client to close during a data transfer, it could be a firewall somewhere closing the TCP connection. My own firewall does this if it finds something nasty part way through the file, it's been deleting mails supposedly from the taxman at gateway.gov.uk today which are actually viruses, the emails are very authentic since I had a real one yesterday when I submitted taxes online, and then got a reply to the email address less than 12 hours later that was almost the same except for a zip attachment. Or maybe the FTP server has a badly implemented session timeout? Can you get the server logs or is not your server? Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
Angus Now trying with your components and although it's much better behaved and fails gracefully, it still fails (in my app or the supplied demo app). However as both your app and my app fail with my hosted FTP server and both work with your FTP server, I'm going to assume that there's something amiss with the ISP's FTP offering. Any clues in the log as to what it could be ? This is the end of the log - though it doesn't always die in the same place - sometimes it's during the directory listing stage. Source Directory: /pub/HireTRACK NX/autoupdates/ Target Directory: c:\FTP_test Source Files Skipped 0 Selected Total Files 1139, Total Size 700,295,024 Started FTP Download from: ftp.navigator.co.uk > CWD /pub/HireTRACK NX/autoupdates/HireTrack NX/3.0.1.151/ < 250 CWD command successful > PWD < 257 "/pub/HireTRACK NX/autoupdates/HireTrack NX/3.0.1.151" is the current directory > MLST HireTrack NX.CAB < modify=20130803144603;perm=adfr;size=12325384;type=file;unique=811U141EC093;UNIX.group=697921;UNIX.mode=0644;UNIX.owner=10749373; HireTrack NX.CAB < 250 End of list Downloading File: /pub/HireTRACK NX/autoupdates/HireTrack NX/3.0.1.151/HireTrack NX.CAB to c:\FTP_test\HireTrack NX\3.0.1.151\HireTrack NX.CAB, Size 12,325,384 Saved File Resume Info c:\ftp_test\hiretrack nx\3.0.1.151\hiretrack nx_cab.ftp > PASV < 227 Entering Passive Mode (205,196,214,163,235,210). 15:16:02:153 ! Data Socket Connect 15:16:02:153 011A0800 Socket handle created 324 15:16:02:169 TWSocket will connect to 205.196.214.163:60370 > RETR HireTrack NX.CAB 15:16:02:325 ! Data Session Connected (Get) < 150 Opening BINARY mode data connection for HireTrack NX.CAB (12325384 bytes) FTP Session Closed 15:16:04:263 011A0800 TCustomWSocket.Shutdown 1 324 15:16:04:263 ! Data Session closed 15:16:04:263 011A0040 TCustomWSocket.Shutdown 1 292 Downloaded File c:\ftp_test\hiretrack nx\3.0.1.151\hiretrack nx_cab.tmp, Size 321,058 Request Failed: Partial File Downloaded, Resume Allowed, Expected File Size 12,325,384, Actual Size 321,058 Download Failed: 500 Control connection closed - Winsock - No Error (#0) No Connection to Server, Cancelled Downloading Files Task Result: Failed No Connection to Server, Cancelled Downloading Files David -- David Rose Navigator Systems Ltd, Network Business Centre, 9 River Street Place, Julian Road, Bath BA1 2RS Main Tel: +44 (0)207 183 0011 Fax: +44 (0)1225 464984 Company Email: da...@navigator.co.uk (Private e-mail on request) CONFIDENTIALITY NOTICE The contents of this e-mail are confidential to the ordinary user of the e-mail address to which it was addressed and may also be privileged. If you are not the addressee of this e-mail you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. If you have received this e-mail in error please e-mail the sender by replying to this message. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
> Angus, trying to change the code to use your components. Does the > TMagFtp component support a CopyType := FCTypeDates ; Sorry, seems like I never supported specific date ranges for FTP which is mostly used to synchronise remote and local directories, or for specific files. > if so, how does one set the dates as fMagFileCopy is declared as > private so these aren't accessible ? You'd have to create new published properties for the component and makes sure they are passed when comparing the directories. Beware none of my demo or current live applications use date ranges, that feature may have got implemented for a private project 10 years ago but has certainly not been tested in recent years. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
Angus, trying to change the code to use your components. Does the TMagFtp component support a CopyType := FCTypeDates ; if so, how does one set the dates as fMagFileCopy is declared as private so these aren't accessible ? UseUTC := true // whether to compare file timestamps in UTC or local time CopyLTime := Now - 365; CopyHiDT:= Now + 1; Regards David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
On 06/11/2013 18:29, Angus Robertson - Magenta Systems Ltd wrote: It's possible you have reached a large directory that is taking a long time to process, and the session is timing out. I got caught myself the other day with a large download that took longer than 60 seconds to compress using Z Mode, and the session closed before the server responds. Try increasing the Abort Timeout to 3 or 4 minutes and retries to 2 or 3 above it. I'm only using the view remote files option, which presumably only gets a directory listing (about 15 files), furthermore it's sometimes works, sometimes doesn't and when it doesn't it doesn't always fail in the same place. :( David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
> I don't think so (though I will double check), as this is happening > about 30 seconds in to the process, I've already iterated through > all the directories and put together a list of files I need to > download and downloaded quite a few of them. Never the less, your log shows a connected event, followed by the prompts for a new connection to a FTP server, so maybe you disconnected earlier and reconnected somehow? > I tested earlier with your DUNmanager as I read that it uses the > TMagFtp and that had a similar problem (though it could be because > I hadn't set something correctly) > I tried the View Remote Files option, with all logging turned on. > After a couple of hundred files it gave up too. > > > PASV > < 227 Entering Passive Mode (205,196,214,163,179,155). > > MLSD /pub/HireTRACK NX/autoupdates/navigator_nxserver/3.0.1.67/ > FTP Session Closed > Directory Listing Abandoned, Not Connected > Directory Listing Abandoned, Failed Response > > QUIT > Task Result: Failed > Request Failed: No Directory Listing It's possible you have reached a large directory that is taking a long time to process, and the session is timing out. I got caught myself the other day with a large download that took longer than 60 seconds to compress using Z Mode, and the session closed before the server responds. Try increasing the Abort Timeout to 3 or 4 minutes and retries to 2 or 3 above it. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
On 06/11/2013 17:29, Angus Robertson - Magenta Systems Ltd wrote: 15:36:17:405 Request MlsdAsync Done. 15:36:17:405 StatusCode = 0 15:36:17:405 LastResponse was : '' 15:36:17:405 Error = 426 () 15:36:17:701 Session Connected, error = 0 15:36:17:701 Request MlsdAsync Done. 15:36:17:701 StatusCode = 220 15:36:17:701 LastResponse was : '220 ProFTPD 1.3.3a Server (My FTP server) [:::205.196.214.163]' You are sending commands out of sequence, the session connected event appears after you've already sent two commands, you then start to download before you've even logged into the server. You also ignore the 426 error and the other out of sequence responses. Looks like you are using async methods and not waiting for them to complete before sending more commands. I don't think so (though I will double check), as this is happening about 30 seconds in to the process, I've already iterated through all the directories and put together a list of files I need to download and downloaded quite a few of them. The process all started here, where you can see the Session Connected, error = 0, the Session Connected message at 15:36:17:701 occured after the error 426 which I am (wrongly) ignoring 15:35:52:762 Logging started 06/11/2013 15:35:52 15:35:52:762 Created FTP Client 15:35:52:762 in DoGetUpdateFiles 15:35:52:762 Passive Mode = TRUE 15:35:52:762 Attempting to connect to FTP 15:35:52:762 connect to ftp 15:35:53:075 Session Connected, error = 0 15:35:53:387 Request ConnectAsync Done. 15:35:53:387 StatusCode = 230 15:35:53:387 LastResponse was : '230 Anonymous access granted, restrictions apply' 15:35:53:387 No error 15:35:53:387 Attempting directory change to: /pub/HireTRACK NX/autoupdates/ My question is whether FTPClient component has this reconnection and retry option already built in (and I just need to activate it by setting a property) or whether I need to code it up myself ? Retrying won't help if you are not waiting for command to complete. The basic FTP component does not automate anything except the login sequence, and does not handle any errors with retries. Instead you should use TMagFtp from: http://www.magsys.co.uk/delphi/magxfer.asp which automates everything and does file and session retries, if requested. Angus I tested earlier with your DUNmanager as I read that it uses the TMagFtp and that had a similar problem (though it could be because I hadn't set something correctly) I tried the View Remote Files option, with all logging turned on. After a couple of hundred files it gave up too. > PASV < 227 Entering Passive Mode (205,196,214,163,179,155). > MLSD /pub/HireTRACK NX/autoupdates/navigator_nxserver/3.0.1.67/ FTP Session Closed Directory Listing Abandoned, Not Connected Directory Listing Abandoned, Failed Response > QUIT Task Result: Failed Request Failed: No Directory Listing David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FTP Client - reconnecting
> For some reason the FTP server I'm connecting to seems to close the > session for no specific reason - here shown with a 426 error, that > currently my code is ignoring and hence I then get a later problem > 530. > > From my FTPClient logs > > 15:36:17:405 local - C:\ProgramData\Navigator Systems\NX SQL > Server\Downloaded Updates\HireTrack NX\3.0.1.169\HireTrack NX.inf > 15:36:17:405 host - /pub/HireTRACK NX/autoupdates/HireTrack > NX/3.0.1.169/HireTrack NX.inf > 15:36:17:405 changing to binary mode > 15:36:17:405 Request MlsdAsync Done. > 15:36:17:405 StatusCode = 0 > 15:36:17:405 LastResponse was : '' > 15:36:17:405 Error = 426 () > 15:36:17:701 Session Connected, error = 0 > 15:36:17:701 Request MlsdAsync Done. > 15:36:17:701 StatusCode = 220 > 15:36:17:701 LastResponse was : '220 ProFTPD 1.3.3a Server (My FTP > server) [:::205.196.214.163]' > 15:36:17:701 No error > 15:36:17:701 C:\ProgramData\Navigator Systems\NX SQL > Server\Downloaded Updates\HireTrack NX\3.0.1.169\HireTrack NX.inf > 15:36:17:701 starting download > 15:36:17:859 Request GetAsync Done. > 15:36:17:859 StatusCode = 530 > 15:36:17:859 LastResponse was : '530 Please login with USER and > PASS' You are sending commands out of sequence, the session connected event appears after you've already sent two commands, you then start to download before you've even logged into the server. You also ignore the 426 error and the other out of sequence responses. Looks like you are using async methods and not waiting for them to complete before sending more commands. > My question is whether FTPClient component has this reconnection > and retry option already built in (and I just need to activate it > by setting a property) or whether I need to code it up myself ? Retrying won't help if you are not waiting for command to complete. The basic FTP component does not automate anything except the login sequence, and does not handle any errors with retries. Instead you should use TMagFtp from: http://www.magsys.co.uk/delphi/magxfer.asp which automates everything and does file and session retries, if requested. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
