Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> I may well have some more questions soon on another little project > based on the Overbyte FTP Server demo program. Just make sure you start with the V7 FTP server this time, there are numerous improvements and new commands over V5 and V6. I run in on my public web site: ftp://www.magsys.co.uk/ and you can even see your own FTP access stats on SQL driven web page. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Back in the olden days the server did not support UTF-8, it just sent ASCII. So as long as you knew what codepage it was using there was not a problem. I'll see how it goes now having implemented the "Download To Stream" idea instead of "Download To String". I may well have some more questions soon on another little project based on the Overbyte FTP Server demo program. Regards Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 08 March 2010 17:34 To: twsocket@elists.org Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > When I get the FEAT response, where do I look for the results? The results are parsed by the component into numerous ftpFeatxxx types so applications know whether specific features are supported. You need to log LastResponse to see them all as literals, but I'm sure your program does that for diagnostics already, otherwise you won't know the reason for commands that fail. > Although I think this is irrelevant since I have found the spec for > the server and the only FEAT supported is UTF8. In which case MLST won't work either. But some FTP servers support special SITE commands that offer alternate indexing that is better than LIST. Also, V2.108 does not support UTF8, that was not added until V7.01, this may not matter provided you never send or receive characters higher than x07F in file or directory names, but otherwise you are in trouble. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
This confirms the documentation I have: Only UTF8 is supported. And I do use : FtpClient1.CodePage := CP_UTF8; As you indicated below. Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Arno Garrels Sent: 08 March 2010 16:16 To: ICS support mailing Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client Graham Powell wrote: > When I get the FEAT response, where do I look for the results? > Although I think this is irrelevant since I have found the spec for > the server and the only FEAT supported is UTF8. if ftpFeatUtf8 in SupportedExtensions then .. if ftpFeatMDTM in SupportedExtensions then .. BTW: I think the OverbyteIcsFtpTst demo doesn't set the UTF-8 code page correctly. Some weeks ago I read in an FTP-RFC (cannot recall the number) that if the UTF-8 Feat is returned by the server and an OPTS UTF8 ON command failed it has to be assumed that UTF-8 is already active on the server-side. To enable UTF-8 by: FtpClient1.CodePage := CP_UTF8; -- Arno Garrels > > Regards > graham > > > -Original Message- > From: twsocket-boun...@elists.org > [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - > Magenta Systems Ltd > Sent: 08 March 2010 14:17 > To: twsocket@elists.org > Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > >> I thought strings were dynamic. The string was always cleared before >> each FTP command. > > Strings are dynamic, but TWSocket returns buffers not strings, and you > normally use MOVE to fill a string, which means increasing the size > manually. > >> I shall investigate your MLST idea. > > Can you post the response to the FEAT command for the newsroom server, > so we can see what commands it supports? And anything that might > identify the FTP server origins? > > Angus > > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> BTW: I think the OverbyteIcsFtpTst demo doesn't set > the UTF-8 code page correctly. Some weeks ago I read > in an FTP-RFC (cannot recall the number) that if the > UTF-8 Feat is returned by the server and an OPTS > UTF8 ON command failed it has to be assumed that UTF-8 > is already active on the server-side. Some servers support UTF8 ON and UTF8 OFF, some only support UFT8 so will turn on support even if you are trying to disable it. Most default OFF, except FileZilla. There are various comments about this in the source code and in the 'Unicode Compatibility with various web servers' comments at: http://www.magsys.co.uk/delphi/magxfer.asp Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> When I get the FEAT response, where do I look for the results? The results are parsed by the component into numerous ftpFeatxxx types so applications know whether specific features are supported. You need to log LastResponse to see them all as literals, but I'm sure your program does that for diagnostics already, otherwise you won't know the reason for commands that fail. > Although I think this is irrelevant since I have found the spec > for the server and the only FEAT supported is UTF8. In which case MLST won't work either. But some FTP servers support special SITE commands that offer alternate indexing that is better than LIST. Also, V2.108 does not support UTF8, that was not added until V7.01, this may not matter provided you never send or receive characters higher than x07F in file or directory names, but otherwise you are in trouble. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Graham Powell wrote: > When I get the FEAT response, where do I look for the results? > Although I think this is irrelevant since I have found the spec for > the server and the only FEAT supported is UTF8. if ftpFeatUtf8 in SupportedExtensions then .. if ftpFeatMDTM in SupportedExtensions then .. BTW: I think the OverbyteIcsFtpTst demo doesn't set the UTF-8 code page correctly. Some weeks ago I read in an FTP-RFC (cannot recall the number) that if the UTF-8 Feat is returned by the server and an OPTS UTF8 ON command failed it has to be assumed that UTF-8 is already active on the server-side. To enable UTF-8 by: FtpClient1.CodePage := CP_UTF8; -- Arno Garrels > > Regards > graham > > > -Original Message- > From: twsocket-boun...@elists.org > [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - > Magenta Systems Ltd > Sent: 08 March 2010 14:17 > To: twsocket@elists.org > Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > >> I thought strings were dynamic. The string was always cleared before >> each FTP command. > > Strings are dynamic, but TWSocket returns buffers not strings, and you > normally use MOVE to fill a string, which means increasing the size > manually. > >> I shall investigate your MLST idea. > > Can you post the response to the FEAT command for the newsroom > server, so we can see what commands it supports? And anything that > might identify the FTP server origins? > > Angus > > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
When I get the FEAT response, where do I look for the results? Although I think this is irrelevant since I have found the spec for the server and the only FEAT supported is UTF8. Regards graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 08 March 2010 14:17 To: twsocket@elists.org Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > I thought strings were dynamic. The string was always cleared before > each FTP command. Strings are dynamic, but TWSocket returns buffers not strings, and you normally use MOVE to fill a string, which means increasing the size manually. > I shall investigate your MLST idea. Can you post the response to the FEAT command for the newsroom server, so we can see what commands it supports? And anything that might identify the FTP server origins? Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> I thought strings were dynamic. The string was always cleared > before each FTP command. Strings are dynamic, but TWSocket returns buffers not strings, and you normally use MOVE to fill a string, which means increasing the size manually. > I shall investigate your MLST idea. Can you post the response to the FEAT command for the newsroom server, so we can see what commands it supports? And anything that might identify the FTP server origins? Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
I have got the MemoryStream idea working although I had to declare the stream as a TMemoryStream otherwise DirStream.Clear will not compile. I thought strings were dynamic. The string was always cleared before each FTP command. I shall investigate your MLST idea. At least I don't have to keep modifying the FTP Client component each time there is a new release. So that's a bonus already. And maybe the way I was implementing the DownloadToString was not as good as it could have been. Many thanks for you help in this matter. Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 05 March 2010 11:11 To: twsocket@elists.org Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > I also mentioned previously that I have modified the FTP Client to > download to a String rather than a file or a Stream. Unless you create a string that is hundreds of thousands of bytes long, or continually extend it each time it's written, you risk overflowing a string since they are not inherently dynamic. > If I ditch the "Download to String" idea and download to a stream, can > I do this without having any disk activity. Is the stream created in > memory or on the disk. A Delphi TStream is an abstract data type, the underlaying format depends upon which specific descendent is created, it may be a memory stream, file stream or various others like buffered streams which are a mix of the two. To save directory listings to a stream, you have something like: var DirStream: TStream ; DirStream := TMemoryStream.Create ; FtpClient.LocalFileName := '' ; FtpClient.LocalStream := DirStream ; FtpClient.Dir ; // process stream DirStream.Free ; FtpClient.LocalStream := nil; Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Hello, Yes I am aware of the grief that can be caused by just assigning an AnsiString to a String and this has been tested thoroughly. FYI: The data from the server is encoded in UTF-8. I don't think the problem here is one of corrupted data, but I get the impression that certain types of network activity is causing me grief. Very difficult to pin down as I have a server here in my office and I have never had a problem. A typical TV station has a very complex network setup with links all over the world. The latest problem was at the Winter Olympics with the server in New York and the FTP client in Vancouver. Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Arno Garrels Sent: 06 March 2010 19:36 To: ICS support mailing Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client Graham Powell wrote: > The directory listing contains a list of story identifiers which look > something like this 123454678:12345678:12345678 The date and time of > the story is one of the fields. So the listing gives me the stories I > need to download and the timestamp. When a story changes I re-download > it. If it disappears from the list I delete it. Since you mentioned converting the response to string with 2010, are you sure that conversion is always properly done (correct code page)? -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Graham Powell wrote: > The directory listing contains a list of story identifiers which look > something like this 123454678:12345678:12345678 > The date and time of the story is one of the fields. So the listing > gives me the stories I need to download and the timestamp. When a > story changes I re-download it. If it disappears from the list I > delete it. Since you mentioned converting the response to string with 2010, are you sure that conversion is always properly done (correct code page)? -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
The directory listing contains a list of story identifiers which look something like this 123454678:12345678:12345678 The date and time of the story is one of the fields. So the listing gives me the stories I need to download and the timestamp. When a story changes I re-download it. If it disappears from the list I delete it. Regards Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 05 March 2010 11:40 To: twsocket@elists.org Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > As I have mentioned in previous posts some months ago I am interfacing > to an Avid iNEWS newsroom server and polling by using the dirAsync > command sent every 3 seconds. Are you checking for changes to a specific file, or just looking for new files in general? If the former, using the MLST or MDTM commands is much more efficient for the server since they don't involve creating a new data socket every three seconds. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> As I have mentioned in previous posts some months ago I am > interfacing to an Avid iNEWS newsroom server and polling by using > the dirAsync command sent every 3 seconds. Are you checking for changes to a specific file, or just looking for new files in general? If the former, using the MLST or MDTM commands is much more efficient for the server since they don't involve creating a new data socket every three seconds. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> I also mentioned previously that I have modified the FTP Client to > download to a String rather than a file or a Stream. Unless you create a string that is hundreds of thousands of bytes long, or continually extend it each time it's written, you risk overflowing a string since they are not inherently dynamic. > If I ditch the "Download to String" idea and download to a stream, > can I do this without having any disk activity. Is the stream > created in memory or on the disk. A Delphi TStream is an abstract data type, the underlaying format depends upon which specific descendent is created, it may be a memory stream, file stream or various others like buffered streams which are a mix of the two. To save directory listings to a stream, you have something like: var DirStream: TStream ; DirStream := TMemoryStream.Create ; FtpClient.LocalFileName := '' ; FtpClient.LocalStream := DirStream ; FtpClient.Dir ; // process stream DirStream.Free ; FtpClient.LocalStream := nil; Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Providing detailed logs is very difficult as the client is NBC in the U.S. As I have mentioned in previous posts some months ago I am interfacing to an Avid iNEWS newsroom server and polling by using the dirAsync command sent every 3 seconds. I also mentioned previously that I have modified the FTP Client to download to a String rather than a file or a Stream. This was to try an eliminate any disk activity when getting directory listings or files. If I ditch the "Download to String" idea and download to a stream, can I do this without having any disk activity. Is the stream created in memory or on the disk. Looking at the code it requires a filename so I assume it is on the disk. I don't have much experience of using streams, so any guidance would be appreciated. I would like to know if this is possible before I embark on a learning experience to no avail. Regards Graham -Original Message- From: twsocket-boun...@elists.org [mailto:twsocket-boun...@elists.org] On Behalf Of Angus Robertson - Magenta Systems Ltd Sent: 04 March 2010 10:31 To: twsocket@elists.org Subject: Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client > Every time I upgrade from the old FPiette software to any of the > Overbyte versions I have been experiencing the occasional problem. What problem? You need to supply detailed logs showing the specific component versions being used, commands sent and received and the errors. > the application basically sends a LIST command to the server every > 3 seconds Not a good idea to poll a server in this way, some firewalls and routers may get upset by continual short connections. I had a cheap Netgear router a long time ago that broke repeated LIST commands. This is one reason I added the FTP CMLSD command to avoid short data connections. > So my question to anybody in the know is this: Is there any > fundamental difference in the way these two FTPClient components work? No. I've had every different version of the FTP client and server components in use daily for several years, with no long term 'occasional problems'. There may have been short term problems in new beta versions which were fixed within a few days. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
Every time I upgrade from the old FPiette software to any of the Overbyte versions So my question to anybody in the know is this: Is there any fundamental difference in the way these two FTPClient components work? These are basically the same components. What you name "Overbyte versions" is ICS-V6 and ICS-V7 where the unit names has been prefixed by "OverbyteIcs". Of course there where evolutions, but basically the component are fundamentally identicals. As Angus said, please provide a diagnostic. Saying you have "occasional problems" do not permit anyone to help you. The component are very stable so it is likely an external problem. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] FPiette V2.108 vs Overbyte V7.07 - FTP Client
> Every time I upgrade from the old FPiette software to any of the > Overbyte versions I have been experiencing the occasional problem. What problem? You need to supply detailed logs showing the specific component versions being used, commands sent and received and the errors. > the application basically sends a LIST command to the server every > 3 seconds Not a good idea to poll a server in this way, some firewalls and routers may get upset by continual short connections. I had a cheap Netgear router a long time ago that broke repeated LIST commands. This is one reason I added the FTP CMLSD command to avoid short data connections. > So my question to anybody in the know is this: Is there any > fundamental difference in the way these two FTPClient components work? No. I've had every different version of the FTP client and server components in use daily for several years, with no long term 'occasional problems'. There may have been short term problems in new beta versions which were fixed within a few days. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
