Re: [twsocket] FTP Client - Large files

2012-12-05 Thread Graham Powell
I have put some more log files on the website:

www.deephaven.co.uk/downloads/ftpuplog.zip

Two files pass (small file) and fail (large file) contain more logging
info.
Not sure if this tells us more about the end of the transfer though.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 04 December 2012 17:33
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 I have the OnFtpStateChange producing messages which I can add to my 
 history log, but is there a way of adding these messages into the ICS 
 logger?

IcsLogger.DoDebugLog(Self, LogOption, Msg);

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-05 Thread RTT



I have put some more log files on the website:

www.deephaven.co.uk/downloads/ftpuplog.zip

Two files pass (small file) and fail (large file) contain more logging
info.
Not sure if this tells us more about the end of the transfer though.



Maybe what you need to do is to keep the control channel from timing out 
(Idle timeout), sending a dummy command from time to time?

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-05 Thread Angus Robertson - Magenta Systems Ltd
 Not sure if this tells us more about the end of the transfer though.

The only thing it shows is the FTP session left unfinished, because your
application does not abort the upload on a timeout, after x seconds of
inactivity.  You update your timer from the OnProgress64 event.  

Now repeat the large upload using xferdemo3, tick high and low level
logging, and create a log file by copying everything from the lower
window.  

Check carefully to see what is different from your own application. 

You could also check for this server bug from the OnProgress64 event, by
seeing if the entire file has been transmitted, calling abort for a clean
close down, then doing a SizeAsync to check the correct sized file is on
the server. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-04 Thread Graham Powell
1:
How do I log what is being sent? I have everything ticked in the ICS logger.

2:
When I use ConnectFeatAsync I get only these 3 things specified in the log:

MDTM
REST: STREAM
SIZE

The other FTP sites have a lot more. Is this a clue?

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 29 November 2012 17:47
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 Files can be found here:

You are not currently logging any commands sent by the FTP component which
is normally quite useful. 

You are not sending the FEAT command which might hint as to which FTP server
is at fault, use ConnectFeatAsync instead of ConnectAsync. 

In the first log, you seem to have an extra shut down for handle 1988 which
appears to be from any unlogged earlier FTP session that was unfinshed
before you started this session.  

12:38:03:331 0322F130 Socket handle created 1992
12:38:03:509 ! Data Session requested
12:38:03:510 0322F9B0 Socket accepted 1944
12:38:03:511 0322F9B0 TCustomWSocket.Shutdown 1 1988
12:38:03:512 ! Data Session opened
12:38:03:514 |150 Opening BINARY mode data connection 

This may be unconnected, but the component should not really know about
earlier handles at this point.  

Angus


--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-04 Thread Angus Robertson - Magenta Systems Ltd
 How do I log what is being sent? I have everything ticked in the 
 ICS logger.

The ICS logger is really designed for low level packet logging, although
I wrote it a long time ago, I rarely use it since the low level stuff
just works. 

MagentaFtp3.pas shows how logging is done, from the events:
onFtpClientDisplay, onFtpError, OnFtpCommand, OnFtpResponse, nFtpSessConn,
OnFtpSessClosed, OnFtpRequestDone and OnFtpStateChange, although my code
is commented out in the last two.  

I did however ask you to log OnFtpStateChange last week, since that is
what causes my transfers to work.  

 When I use ConnectFeatAsync I get only these 3 things specified in 
 the log:
 
 MDTM
 REST: STREAM
 SIZE

It's a relatively primitive FTP server, not from Microsoft, no support
for UTF8, most server written in the last 10 years support more optional
FTP features. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-04 Thread Graham Powell
I have the OnFtpStateChange producing messages which I can add to my history
log, but is there a way of adding these messages into the ICS logger?

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 04 December 2012 15:52
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 How do I log what is being sent? I have everything ticked in the ICS 
 logger.

The ICS logger is really designed for low level packet logging, although I
wrote it a long time ago, I rarely use it since the low level stuff just
works. 

MagentaFtp3.pas shows how logging is done, from the events:
onFtpClientDisplay, onFtpError, OnFtpCommand, OnFtpResponse, nFtpSessConn,
OnFtpSessClosed, OnFtpRequestDone and OnFtpStateChange, although my code is
commented out in the last two.  

I did however ask you to log OnFtpStateChange last week, since that is what
causes my transfers to work.  

 When I use ConnectFeatAsync I get only these 3 things specified in the 
 log:
 
 MDTM
 REST: STREAM
 SIZE

It's a relatively primitive FTP server, not from Microsoft, no support for
UTF8, most server written in the last 10 years support more optional FTP
features. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-12-04 Thread Angus Robertson - Magenta Systems Ltd
 I have the OnFtpStateChange producing messages which I can add to 
 my history log, but is there a way of adding these messages into
 the ICS logger?

IcsLogger.DoDebugLog(Self, LogOption, Msg);

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-29 Thread Graham Powell
Attached is a zipped up text file obtained from the ICS logger showing the
upload of a 84MByte file to the FTP server.

No ftpPutAsync generated at the end of the transfer.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 28 November 2012 18:51
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 Filezilla seems to handle with a no response timeout, followed by a 
 query, some times even a  reconnect, to check if the file is 
 effectively there, and with the correct size.

Similar features are the benefit of using the TMagFtp high level
component.   It also checks the correct sized file has been uploaded,
will do a CRC32 or MD5 check to ensure it's not been corrupted, and  will
repeat the upload if necessary.  

But I'm not aware TMagFtp specifically handles a missing response, unless by
accident!  Most servers will timeout the control connection anyway once
activity ceases that could cause TMagFtp to close everything down.
Without having seen any proper FTP logs for this bad server, this is all
conjecture. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] FTP Client - Large files

2012-11-29 Thread Graham Powell
And this attachment is what the ICS logger shows for a smaller file.
At the end of upload a further file is uploaded. Not done in the previous
log as the first transfer failed to complete.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 28 November 2012 18:51
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 Filezilla seems to handle with a no response timeout, followed by a 
 query, some times even a  reconnect, to check if the file is 
 effectively there, and with the correct size.

Similar features are the benefit of using the TMagFtp high level
component.   It also checks the correct sized file has been uploaded,
will do a CRC32 or MD5 check to ensure it's not been corrupted, and  will
repeat the upload if necessary.  

But I'm not aware TMagFtp specifically handles a missing response, unless by
accident!  Most servers will timeout the control connection anyway once
activity ceases that could cause TMagFtp to close everything down.
Without having seen any proper FTP logs for this bad server, this is all
conjecture. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] FTP Client - Large files

2012-11-29 Thread Angus Robertson - Magenta Systems Ltd
 Attached is a zipped up text file obtained from the ICS logger 

The mailing list automatically drops file attachments since they are of
no interest to 99% of recipients. 

You need to email directly to me or post a link to a download. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-29 Thread Graham Powell
Files can be found here:

A: www.deephaven.co.uk/downloads/uploadlogA.zip

B: www.deephaven.co.uk/downloads/uploadlogB.zip

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 28 November 2012 18:51
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 Filezilla seems to handle with a no response timeout, followed by a 
 query, some times even a  reconnect, to check if the file is 
 effectively there, and with the correct size.

Similar features are the benefit of using the TMagFtp high level
component.   It also checks the correct sized file has been uploaded,
will do a CRC32 or MD5 check to ensure it's not been corrupted, and  will
repeat the upload if necessary.  

But I'm not aware TMagFtp specifically handles a missing response, unless by
accident!  Most servers will timeout the control connection anyway once
activity ceases that could cause TMagFtp to close everything down.
Without having seen any proper FTP logs for this bad server, this is all
conjecture. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-29 Thread Angus Robertson - Magenta Systems Ltd
 Files can be found here:

You are not currently logging any commands sent by the FTP component
which is normally quite useful. 

You are not sending the FEAT command which might hint as to which FTP
server is at fault, use ConnectFeatAsync instead of ConnectAsync. 

In the first log, you seem to have an extra shut down for handle 1988
which appears to be from any unlogged earlier FTP session that was
unfinshed before you started this session.  

12:38:03:331 0322F130 Socket handle created 1992
12:38:03:509 ! Data Session requested
12:38:03:510 0322F9B0 Socket accepted 1944
12:38:03:511 0322F9B0 TCustomWSocket.Shutdown 1 1988
12:38:03:512 ! Data Session opened
12:38:03:514 |150 Opening BINARY mode data connection 

This may be unconnected, but the component should not really know about
earlier handles at this point.  

Angus


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-28 Thread Graham Powell
I am trying to see as much information as possible to get an idea of where
the problem lies.

It would appear that this FTP site does not send a FTP 226 Transfer
Complete for large files. So in this case you cannot use the ftpPutAsync
event to know when the upload has finished.

I have used Wireshark to monitor what is sent when you just drag  drop the
file to the FTP server using Windows Explorer. Again there is no Transfer
Complete, but everything works fine.

1: Can the FTP client be modified to generate the ftpPutAsync event on
something else as well as the Transfer Complete. Wireshark always shows some
message at the end that contains FIN ACK.

2: Is there something special about putAsync that makes it different to
just Put with regard to generating the final event.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Arno Garrels
Sent: 27 November 2012 19:22
To: ICS support mailing
Subject: Re: [twsocket] FTP Client - Large files

Angus Robertson - Magenta Systems Ltd wrote:
 Not sure why you using Wireshark, the FTP component has all the events 
 you need to log the FTP protocol, even without the logger component.

I think he uses Wireshark because, and that makes sense, he wants to know
whether or not the server response to the PUT request is actually received.
From how I read OP's messages the response to the PUT request is sometimes
and sometimes not included in the Wireshark logs, which is at least strange.

--
Arno 
   
--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-28 Thread Angus Robertson - Magenta Systems Ltd
 It would appear that this FTP site does not send a FTP 226 Transfer
 Complete for large files. 

So it's a faulty FTP server, not a problem with the FTP client.  

I've already asked what make of FTP software and am waiting for your
response.  You have not posted any FTP logs (the full FTP session) which
might identify it. 

 is there something special about putAsync th

I explained all that yesterday. 

Angus



--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-28 Thread Arno Garrels

- Original Message - 
From: Angus Robertson - Magenta Systems Ltd an...@magsys.co.uk
To: twsocket@elists.org
Sent: Wednesday, November 28, 2012 11:36 AM
Subject: Re: [twsocket] FTP Client - Large files


 It would appear that this FTP site does not send a FTP 226 Transfer
 Complete for large files. 
 
 So it's a faulty FTP server, not a problem with the FTP client.

Or only that server is behind a router or it is the slowest. I still think
it's likely a faulty router that times out the control connection, hence the
FTP 226 Transfer Complete response is very likely sent however dropped by
the router, so not logged in Wireshark.
  
-- 
Arno
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-28 Thread RTT

On 28-11-2012 08:51, Graham Powell wrote:

1: Can the FTP client be modified to generate the ftpPutAsync event on
something else as well as the Transfer Complete. Wireshark always shows some
message at the end that contains FIN ACK.


Sometimes, when I'm updating my web site using Filezilla, and specially 
with large files, I think I also experience this lack of confirmation of 
transfer complete, that Filezilla seems to handle with a no response 
timeout, followed by a query, some times even a reconnect, to check if 
the file is effectively there, and with the correct size. I'm not used 
to the FTP protocol, so never gave attention to the Filezilla activity 
log, to see what's exactly happening, but that's the idea I get.

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-28 Thread Angus Robertson - Magenta Systems Ltd
 Filezilla seems to handle with a no response timeout, followed by a
 query, some times even a  reconnect, to check if the file is
 effectively there, and with the correct size. 

Similar features are the benefit of using the TMagFtp high level
component.   It also checks the correct sized file has been uploaded,
will do a CRC32 or MD5 check to ensure it's not been corrupted, and  will
repeat the upload if necessary.  

But I'm not aware TMagFtp specifically handles a missing response, unless
by accident!  Most servers will timeout the control connection anyway
once activity ceases that could cause TMagFtp to close everything down.
Without having seen any proper FTP logs for this bad server, this is all
conjecture. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-27 Thread Graham Powell
More clues.

1:
Tried the xferdemo3 and that works ok.

2:
My code is basically a state machine. In the FtpRequestDone event, it
decides what to do next. So after a CWD and a possible Mkdir commands it
does the putAsync. Nothing then happens until the next event which should be
a ftpPutAsync. However in the case of the large file and this particular FTP
site, no more events are generated.

If I change the putAsync to Put,  (again from within the FtpRequestDone
event) the transfer is performed as before. Now at the end of the transfer
the code continues to the next line after the Put as you would expect.
Next, a ftpPutAsync event is generated with an error code of 426.

Wireshark is still not showing 226 File Transfer Complete.

3:
Modified the xferdemo3 code by adding a FtpRequestDone event just to see
what and when it is called. This behaves as I expect and the ftpPutAsync is
generated at the end of the transfer. But I assume this is because the FTP
server has sent a 226 FTP Transfer Complete.

So the question is: What are you doing in your demo that is different.
Wireshark shows you are doing ALLO, but that just gets a response saying
it's unnecessary.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 26 November 2012 18:31
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 My code just does putAsync and waits for the ftpPutAsync event.
 I don't know exactly how the xferdemo3w does it

You have all the source code for TMagFtp, so you can easily check.  There
are probably two main differences: 

1 - TMagFtp uses the sync FTP calls and WaitUntilReady, so check that
function for special handling.  

2 - The component has been developed and tested over several years and so is
very solid with a lot of special error handling to make it very stable.


One caution, xferdemo3w uses a 'special' widestring version of ICS v8 that
works with Delphi 2007, to use the unchanged ICS version test against
xferdemo3 which is pure ICS. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-27 Thread Angus Robertson - Magenta Systems Ltd
 My code is basically a state machine. In the FtpRequestDone event, 
 it decides what to do next. 

I used a similar state machine in another application, but prefer the
sync calls for complex applications, and then use a thread per transfer.
TMagFtp has been tested with 200 simultaneous downloads. 

You do have to be very careful not to call the message pump from within
RequestDone or things can get very confused.  Do not call any sync
methods that will call the message pump, ie only call CwdAsync not Cwd. 

 So the question is: What are you doing in your demo that is 
 different.

Using sync methods with WaitUntilReady which loops calling the message
pump until State is ftpReady or ftpInternalReady, rather than expecting
the FtpRequestDone event to be called. 

It is possible there is a strange component bug and the state is being
changed without FtpRequestDone being called.  My component does not use
FtpRequestDone atall.  

You could try checking the StateChange event as well to see if that is
called when your transfer finishes, or just check the State property in
the fail safe timer I assume you have running to handle errors and stop
the application waiting for days. 

 Wireshark shows you are doing ALLO

Not sure why you using Wireshark, the FTP component has all the events
you need to log the FTP protocol, even without the logger component.  

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-27 Thread Arno Garrels
Angus Robertson - Magenta Systems Ltd wrote:
 Not sure why you using Wireshark, the FTP component has all the events
 you need to log the FTP protocol, even without the logger component.

I think he uses Wireshark because, and that makes sense, he wants 
to know whether or not the server response to the PUT request is actually
received. From how I read OP's messages the response to the PUT request
is sometimes and sometimes not included in the Wireshark logs, which is 
at least strange.

-- 
Arno 
   
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-26 Thread Graham Powell
More clues.

Now looking at Wireshark data to try and analyse what is going on.

Test 1: 
My application small FTP file: All OK

Test 2:
My application with large file: Not OK, seem to be missing the FTP 226
Transfer Complete from server to client.

Test3:
Running xferdemo3w. This works OK. The FTP 226 message is present in
Wireshark.

So what can be different?

My code just does putAsync and waits for the ftpPutAsync event.
I don't know exactly how the xferdemo3w does it, but it must be doing
something differently.

I have tried Passive, Active, ASCII and Binary modes - no difference.

Graham


-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 24 November 2012 14:05
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 *** So the duff FTP site will accept the upload of the file, but for 
 some reason the FTP client does not generate any event
 (FTPRequestDone) at the end of the upload.

And yet you said earlier that the TMagFtp component works correctly,
presumably with the same bad FTP server and ICS version?  

Do you know the software used on the bad server?  I have a public FTP server
with several different FTP servers, including ICS at ics.ftptest.co.uk. 

Angus


--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-26 Thread Arno Garrels
Graham Powell wrote:
 
 I have tried Passive, Active, ASCII and Binary modes - no difference.

Have you tried with the OverbyteIcsFtpTst demo yet?

-- 
Arno Garrels

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-26 Thread Graham Powell
The OverbyteIcsFtpTst demo suffers from the same problem. No ftpPutAsync
event generated at the end of the transfer.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Arno Garrels
Sent: 26 November 2012 15:49
To: ICS support mailing
Subject: Re: [twsocket] FTP Client - Large files

Graham Powell wrote:
 
 I have tried Passive, Active, ASCII and Binary modes - no difference.

Have you tried with the OverbyteIcsFtpTst demo yet?

--
Arno Garrels

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-26 Thread Arno Garrels
Graham Powell wrote:
 The OverbyteIcsFtpTst demo suffers from the same problem. No
 ftpPutAsync event generated at the end of the transfer.

Interesting, Angus should know how he tweaks the ICS component
to make them working.

-- 
Arno

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-26 Thread Angus Robertson - Magenta Systems Ltd
 My code just does putAsync and waits for the ftpPutAsync event.
 I don't know exactly how the xferdemo3w does it

You have all the source code for TMagFtp, so you can easily check.  There
are probably two main differences: 

1 - TMagFtp uses the sync FTP calls and WaitUntilReady, so check that
function for special handling.  

2 - The component has been developed and tested over several years and so
is very solid with a lot of special error handling to make it very stable.


One caution, xferdemo3w uses a 'special' widestring version of ICS v8
that works with Delphi 2007, to use the unchanged ICS version test
against xferdemo3 which is pure ICS. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-24 Thread Graham Powell
Moe info.

As I said in a previous e-mail. I am testing with 3 FTP sites. It appear
only one of them has this problem.
Below is the extract from the ICS Logger at the end of the upload for the
84MByte file:

This is end of the ICS logger when I upload to the failing FTP site:

12:17:42:193 DataSocketPutDataSent 32768
12:17:42:350 03C1F9B0 TriggerDataSent 1800
12:17:42:358 DataSocketPutDataSent 23974
12:17:42:468 03C1F9B0 TriggerDataSent 1800
12:17:42:470 DataSocketPutDataSent 0
12:17:42:473 03C1F9B0 TCustomWSocket.Shutdown 1 1800
12:17:42:815 ! Data Session closed
12:17:42:817 ! Next3PutAsync
12:17:42:820 03C1F9B0 TCustomWSocket.Shutdown 1 1800   *** THIS IS THE LAST
MESSAGE IN THE LOGGER ***

This the end of the upload to an FTP site that is OK:

12:52:36:382 DataSocketPutDataSent 32768
12:52:36:624 03B6F9B0 TriggerDataSent 1824
12:52:36:627 DataSocketPutDataSent 32768
12:52:36:870 03B6F9B0 TriggerDataSent 1824
12:52:36:882 DataSocketPutDataSent 23974
12:52:37:067 03B6F9B0 TriggerDataSent 1824
12:52:37:077 DataSocketPutDataSent 0
12:52:37:080 03B6F9B0 TCustomWSocket.Shutdown 1 1824
12:52:37:138 ! Data Session closed
12:52:37:140 ! Next3PutAsync
12:52:37:143 03B6F9B0 TCustomWSocket.Shutdown 1 1824
12:52:37:196 |226 File receive OK.|  *** THIS IS MISSING IN THE OTHER FTP
UPLOAD ***

12:52:37:198 ! Next3PutAsync   *** THIS IS THE START OF THE NEXT UPLOAD ***
12:52:37:200 ! HighLevelAsync 0
12:52:37:201 ! HighLevelAsync done
12:52:37:204 ! HighLevelAsync 0
12:52:37:207 03B6F130 TriggerDataSent 1832
12:52:37:263 |200 PORT command successful. Consider using PASV.|

12:52:37:264 ! HighLevelAsync 0
12:52:37:265 03B6F130 TriggerDataSent 1832
12:52:37:382 ! Data Session requested
12:52:37:387 03B6F9B0 Socket accepted 756
12:52:37:393 03B6F9B0 TCustomWSocket.Shutdown 1 1512
12:52:37:397 ! Data Session opened
12:52:37:400 |150 Ok to send data.|

12:52:37:404 03B6F9B0 TriggerDataSent 756
12:52:37:407 DataSocketPutDataSent 32768
12:52:37:742 03B6F9B0 TriggerDataSent 756
12:52:37:751 DataSocketPutDataSent 32768
12:52:37:989 03B6F9B0 TriggerDataSent 756
12:52:38:000 DataSocketPutDataSent 32768

*** So the duff FTP site will accept the upload of the file, but for some
reason the FTP client does not generate any event (FTPRequestDone) at the
end of the upload.

Graham



-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 23 November 2012 16:37
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 I made a typo with the file size. It is 84Mbytes (not 84GBytes) that 
 fails. 39MByte file is OK

Those are trivial sizes for FTP uploads, no reason for difference in
behaviour between those sizes. 

Is this file specific?  What about 100 or 200 megs?  

When I was downloading a VHD (disk image) file, the transfer kept failing at
about 6 gigs (and TMagFtp then repeated the download several times). I
finally discovered by hardware firewall was virus checking FTP downloads and
found a development tool on the disk image that it considered a hacking
tool, so closed the FTP connection.  So can waste time looking for a bug
that is actually totally unrelated to your code. 

Basically you need a lot of logging, OnRequestDone in particular, also
OnStateChange, to find out what events are called normally, and then see
what is missing when it fails.   

 Your Mag demo works.

So nothing wrong with ICS FTP, it must be your code.  Why not use TMagFtp,
it works and hides all this hard stuff from you. 

 I have tried playing with the FTP Client Timeout property, but I don't 
 think this is relevant here.

Timeout is only used when calling sync events, so that the function calls
return at some point.  

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-24 Thread Arno Garrels
Graham Powell wrote:
 12:52:37:143 03B6F9B0 TCustomWSocket.Shutdown 1 1824
 12:52:37:196 |226 File receive OK.|  *** THIS IS MISSING IN THE
 OTHER FTP UPLOAD ***


http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html#FirewallTimeouts
Comes to mind, however I wonder why it happens only with one FTP server, 
maybe it is the slowest? Or that server is behind a router.

-- 
Arno

 
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-24 Thread Fran├žois Piette
 As I said in a previous e-mail. I am testing with 3 FTP sites. It appear
only one of them has this problem.

This probably means the problem is at the server side.
Please try with another FTP client, for example windows own command line FTP
client. Use the same file and do that from the same computer, using same
mode (binary/text, active/passive).
Also try with another file just in case it depends on the file content.


-- 
francois.pie...@overbyte.be
Embarcadero MVP
http://www.overbyte.be




--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-24 Thread Angus Robertson - Magenta Systems Ltd
 *** So the duff FTP site will accept the upload of the file, but 
 for some reason the FTP client does not generate any event
 (FTPRequestDone) at the end of the upload.

And yet you said earlier that the TMagFtp component works correctly,
presumably with the same bad FTP server and ICS version?  

Do you know the software used on the bad server?  I have a public FTP
server with several different FTP servers, including ICS at
ics.ftptest.co.uk. 

Angus


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-23 Thread Graham Powell
Is the a limit to the size of file I can PutAsync?

I am trying to upload a large file (84GBytes) to a FTP site. The file seems
to transfer OK, but there is nothing to signal the transfer is complete. No
ftpPutAsync and no ftpError event.

Smaller files are OK.

Graham



--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-23 Thread Angus Robertson - Magenta Systems Ltd
 Is the a limit to the size of file I can PutAsync?

No limits in recent versions of ICS that use int64.  
 
 I am trying to upload a large file (84GBytes) to a FTP site. The 
 file seems
 to transfer OK, but there is nothing to signal the transfer is 
 complete. No ftpPutAsync and no ftpError event.

I've certainly transferred VHD files in the 10 to 20 gig region, but
never 84 gigs.  Since ICS is event driven, when the data and control
sockets close you must be getting events of some sort, irrespective of
file size or transfer time.

The most common reason for errors like this are your application has a
timer that has not been regular updated during the transfer process, and
has given up waiting for the transfer to finish.  

I suggest you try the compiled demo application from TMagFtp which has a
Single FTP option and see it that works OK, it should. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-23 Thread Graham Powell
1:
I am using ICS V8.

2:
I made a typo with the file size. It is 84Mbytes (not 84GBytes) that fails.
39MByte file is OK

3:
I have no timers running that could mess things up.

4:
Your Mag demo works.

5:
I have tried playing with the FTP Client Timeout property, but I don't think
this is relevant here.

6:
The OnProgress64 event works right up to the 84Mbyte end point.

Any more clues.

Graham

-Original Message-
From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On
Behalf Of Angus Robertson - Magenta Systems Ltd
Sent: 23 November 2012 13:55
To: twsocket@elists.org
Subject: Re: [twsocket] FTP Client - Large files

 Is the a limit to the size of file I can PutAsync?

No limits in recent versions of ICS that use int64.  
 
 I am trying to upload a large file (84GBytes) to a FTP site. The file 
 seems to transfer OK, but there is nothing to signal the transfer is 
 complete. No ftpPutAsync and no ftpError event.

I've certainly transferred VHD files in the 10 to 20 gig region, but never
84 gigs.  Since ICS is event driven, when the data and control sockets close
you must be getting events of some sort, irrespective of file size or
transfer time.

The most common reason for errors like this are your application has a timer
that has not been regular updated during the transfer process, and has given
up waiting for the transfer to finish.  

I suggest you try the compiled demo application from TMagFtp which has a
Single FTP option and see it that works OK, it should. 

Angus

--
To unsubscribe or change your settings for TWSocket mailing list please goto
http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] FTP Client - Large files

2012-11-23 Thread Angus Robertson - Magenta Systems Ltd
 I made a typo with the file size. It is 84Mbytes (not 84GBytes) 
 that fails. 39MByte file is OK

Those are trivial sizes for FTP uploads, no reason for difference in
behaviour between those sizes. 

Is this file specific?  What about 100 or 200 megs?  

When I was downloading a VHD (disk image) file, the transfer kept failing
at about 6 gigs (and TMagFtp then repeated the download several times). I
finally discovered by hardware firewall was virus checking FTP downloads
and found a development tool on the disk image that it considered a
hacking tool, so closed the FTP connection.  So can waste time looking
for a bug that is actually totally unrelated to your code. 

Basically you need a lot of logging, OnRequestDone in particular, also
OnStateChange, to find out what events are called normally, and then see
what is missing when it fails.   

 Your Mag demo works.

So nothing wrong with ICS FTP, it must be your code.  Why not use TMagFtp,
it works and hides all this hard stuff from you. 

 I have tried playing with the FTP Client Timeout property, but I 
 don't think this is relevant here.

Timeout is only used when calling sync events, so that the function calls
return at some point.  

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be