Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Arno Garrels
Angus Robertson - Magenta Systems Ltd wrote:
 I'm writing simple console downloader and decided to use sync
 operations in order to simplify structure. But I've found that
 sync requests (HttpCli) do not use any timeout, so they may last
 forever in case of error.
 
 That's IMHO a bug.
 
 Indeed, poor implementation in THttpCli with no timeout.
 
 But most components do have a timeout for sync methods, for instance
 TSyncSmtpCli and TFtpClient.

Yes they have, but do they work properly when a send or receive lasts 
longer then the timeout value? 

Please guys, take a look at this fix for V7 and test it, anything wrong
with it?
 
{code}
procedure THttpCli.DoRequestSync(Rq : THttpRequest);
var
DummyHandle : {$IFDEF CLR}Borland.Vcl.Windows.THandle;
  {$ELSE} THandle;{$ENDIF}
TimeOutMsec : UINT;
begin
DoRequestAsync(Rq);
if not Assigned(FCtrlSocket.Counter) then
FCtrlSocket.CreateCounter;
FCtrlSocket.Counter.SetConnected; // Reset counter
DummyHandle := INVALID_HANDLE_VALUE;
TimeOutMsec := 5000;//FTimeOut * 1000; // hardcoded 5 sec for testing only
while FState  httpReady do
begin
if MsgWaitForMultipleObjects(0, DummyHandle, FALSE, 1000,
 QS_ALLINPUT) = WAIT_OBJECT_0 then
FCtrlSocket.MessagePump;
if (FState  httpReady) and (
   {$IFNDEF NOFORMS} Application.Terminated or {$ENDIF} FTerminated or
   (IcsCalcTickDiff(FCtrlSocket.Counter.LastAliveTick,
GetTickCount) = TimeOutMsec)) then
begin
Abort;
break;
end;
end;

{* Jul 12, 2004
   WARNING: The component now doesn't consider 401 status
as a fatal error (no exception is triggered). This required a
change in the application code if it was using the exception that
is no more triggered for status 401 and 407.
*}
{* if FStatusCode  401 thenDec 14, 2004 *}
if (FStatusCode = 400) and (FStatusCode  401) and (FStatusCode  407) 
then
raise EHttpException.Create(FReasonPhrase, FStatusCode);
end;
{code}
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Angus Robertson - Magenta Systems Ltd
  But most components do have a timeout for sync methods, for 
  instance TSyncSmtpCli and TFtpClient.
 
 Yes they have, but do they work properly when a send or receive 
 lasts longer then the timeout value? 

I fixed the timeout in TSyncSmtpCli eight years ago, one of the first
changes I made to ICS.  Sync TFtpClient certainly works as well, I use it
all the time, ditto POP3.  I also use sync THttpCli and did not notice a
timeout missing, but I have my own abort timers anyway.  

Angus



--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Arno Garrels
Arno Garrels wrote:
 Please guys, take a look at this fix for V7 and test it, anything
 wrong with it?

Just noticed that it still lacks a timeout status.

--
Arno Garrels   
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Anton Sviridov
Arno Garrels wrote
TWSocketCounter has been added to TWSocket in V7 with properties 
ConnectTick, ConnectDT, LastAliveTick, LastRecvTick and LastSendTick.
Yes, I'm using it in my FTP class, but Start/StopTimer seem easier in my case

SZ wrote:
I think Arno and Francois is right about not making a TTimer per Thttpcli as
default. It would overwhelm our reverse proxy with 10k connections each with
two sockets. However for simple needs, Anton is right about asking for a
solution which I think sould only be switched on by a define.
The timer could be created only by special call, like it's done for Counters... 
also there are no such necessary in timers in server components, they are 
content with a single one to trace inactive connections. And I doubt that 
anybody will have so much client sockets that he'll run out of Windows 
resources.

Arno Garrels wrote
Please guys, take a look at this fix for V7 and test it, anything wrong
with it?
It works, but I think Abort isn'tt a very good way to tell that there was 
timeout... and 404 even stranger. 404 means that file not exist (constant 
error), Connection aborted on request means that user or control proc decided 
to break the waiting, and timeout means that there's no answer from server and 
we should wait and retry later.

-- 
Anton
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Arno Garrels
Anton Sviridov wrote:

 Arno Garrels wrote
 Please guys, take a look at this fix for V7 and test it, anything
 wrong with it?
 It works, but I think Abort isn'tt a very good way to tell that there
 was timeout... 

Abort on timeout is commonly used in ICS.

 and 404 even stranger. 404 means that file not exist
 (constant error), 

Any alternates (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes)? 

It's not a constant error AFAIK. IMO 408 was more confusing. 
404 Not Found
The requested resource could not be found but may be available again
in the future. Subsequent requests by the client are permissible.

408 Request Timeout
The server timed out waiting for the request.

Connection aborted on request means that user or
 control proc decided to break the waiting, and timeout means that
 there's no answer from server and we should wait and retry later.

OK, what about this one?

{code}
procedure THttpCli.DoRequestSync(Rq : THttpRequest);
var
DummyHandle : {$IFDEF CLR}Borland.Vcl.Windows.THandle;
  {$ELSE} THandle;{$ENDIF}
TimeOutMsec : UINT;
bFlag   : Boolean;
begin
DoRequestAsync(Rq);
if not Assigned(FCtrlSocket.Counter) then
FCtrlSocket.CreateCounter;
FCtrlSocket.Counter.SetConnected; // Reset counter
DummyHandle := INVALID_HANDLE_VALUE;
TimeOutMsec := 5000;//FTimeOut * 1000; // hardcoded 5 sec for testing only
while FState  httpReady do
begin
if MsgWaitForMultipleObjects(0, DummyHandle, FALSE, 1000,
 QS_ALLINPUT) = WAIT_OBJECT_0 then
FCtrlSocket.MessagePump;
if (FState  httpReady) and (
   {$IFNDEF NOFORMS} Application.Terminated or {$ENDIF} FTerminated or
   (IcsCalcTickDiff(FCtrlSocket.Counter.LastAliveTick,
GetTickCount) = TimeOutMsec)) then
begin
bFlag := (FState = httpDnsLookup);
StateChange(httpAborting);

if bFlag then
try
FCtrlSocket.CancelDnsLookup;
except
{ Ignore any exception }
end;
FStatusCode := 404;
if {$IFNDEF NOFORMS} Application.Terminated or {$ENDIF} FTerminated 
then
begin
FReasonPhrase := 'Request aborted';
FRequestDoneError := httperrAborted;
end
else begin
FReasonPhrase := 'Request aborted on timeout';
FRequestDoneError := 8; //new const will be added 
httperrCustomTimeOut = 8
end;
if bFlag then
SocketSessionClosed(Self, 0)
else
FCtrlSocket.Close;
StateChange(httpReady);
break;
end;
end;

{* Jul 12, 2004
   WARNING: The component now doesn't consider 401 status
as a fatal error (no exception is triggered). This required a
change in the application code if it was using the exception that
is no more triggered for status 401 and 407.
*}
{* if FStatusCode  401 thenDec 14, 2004 *}
if (FStatusCode = 400) and (FStatusCode  401) and (FStatusCode  407) 
then
raise EHttpException.Create(FReasonPhrase, FStatusCode);
end;
{code}

--
Arno Garrels

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Anton Sviridov
Any alternates (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes)? 
It's not a constant error AFAIK. IMO 408 was more confusing. 

Yeah, that's why transport level stuff should be done in transport (socket) 
classes :) We would have OnConnect(Request)Timeout or something like and no 
mess with statuses...

OK, what about this one?

That's pretty better, although OnRequestDone is never called and hence 
FRequestDoneError isn't used. Maybe make it readable property?

-- 
Anton
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Arno Garrels
Anton Sviridov wrote:
 Any alternates
 (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes)? It's not a
 constant error AFAIK. IMO 408 was more confusing. 
 
 Yeah, that's why transport level stuff should be done in transport
 (socket) classes :) We would have OnConnect(Request)Timeout or
 something like and no mess with statuses...

Connect-timeouts are not the only possible ones. If you need
different timeout values for different states it has to be
implemented using a timer. We need to keep the message pump
as fast as possible.  
  
 
 OK, what about this one?
 
 That's pretty better, although OnRequestDone is never called and
 hence FRequestDoneError isn't used. Maybe make it readable property? 

Actually it _is called on StateChange(httpReady).

GET Failed !
StatusCode   = 404
ReasonPhrase = Request aborted on timeout
RequestDone Error = 8. Status = 404

--
Arno Garrels
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Anton Sviridov
 Connect-timeouts are not the only possible ones.
Yes, I know, but connect ones are the most general and used by everyone, 
because w/o connection there's nothing to do anyway.

 If you need different timeout values for different states it has to be
 implemented using a timer. We need to keep the message pump
 as fast as possible.
But what about convenience? And every programmer implements these timeouts 
anyway. Or he hopes on ICS and doesnt do this receiving endless connect 
waiting...

Actually it _is called on StateChange(httpReady).
Yes, but it is called with ErrCode = 0 and StatusCode = 0. Also I have strange 
things here:

log of my program, trying to download http://test1.ru/1.php, which is my script 
with endless loop to simulate timeout:

 URL http://test1.ru/1.php : retrieving header... (attempt 1/3) // 
FHttpCli.Head called
tick... 1 // this from my timer, which does nothing but counting
tick... 2
tick... 3
tick... 4
tick... 5
Error: HEAD Failed (404, Request aborted on timeout)! // exception from Head, 
everything's OK
 URL http://test1.ru/1.php : retrieving header... (attempt 2/3) // 
FHttpCli.Head called 2nd time
Done.// HttpRequestDone with ErrCode = 0!
Done.// HttpRequestDone with ErrCode = 0!
Header received, StatusCode = 0 // Head successfully passed!

with my timer everything's OK:

 URL http://test1.ru/1.php : retrieving header... (attempt 1/3)
tick... 1
tick... 2
tick... 3
tick... 4
tick... 5
Failed, error #3
Error: HEAD Failed (404, Connection aborted on request)!
 URL http://test1.ru/1.php : retrieving header... (attempt 2/3)
tick... 1
tick... 2
tick... 3
tick... 4
tick... 5
Failed, error #3
Error: HEAD Failed (404, Connection aborted on request)!
 URL http://test1.ru/1.php : retrieving header... (attempt 3/3)
tick... 2
tick... 3
tick... 4
tick... 5
Failed, error #3
Error: HEAD Failed (404, Connection aborted on request)!
*** finish ***

*

BTW, I have noticed that Last-Modified filed in header isn't parsed! IMHO it's 
too important field to leave parsing of it. Maybe Magenta's code could be added 
for this purpose?

-- 
Anton
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-17 Thread Arno Garrels
Anton Sviridov wrote:
 Connect-timeouts are not the only possible ones.
 Yes, I know, but connect ones are the most general and used by
 everyone, because w/o connection there's nothing to do anyway. 
 
 If you need different timeout values for different states it has to
 be implemented using a timer. We need to keep the message pump
 as fast as possible.
 But what about convenience? And every programmer implements these
 timeouts anyway. Or he hopes on ICS and doesnt do this receiving
 endless connect waiting...  
 
 Actually it _is called on StateChange(httpReady).
 Yes, but it is called with ErrCode = 0 and StatusCode = 0. Also I
 have strange things here: 
 
 log of my program, trying to download http://test1.ru/1.php, which is
 my script with endless loop to simulate timeout: 

[..]

This has been discussed many times before. The sync methods are not
finished until they return, OnRequestDone actually fires before the 
sync method returns!
So either use async methods or post a custom message from OnRequestDone
handler and call the next method from the custom message handler.

--
Arno Garrels


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Anton Sviridov
 how can I add timeouts without messageloops and threads and so on?
I see that there is still ControlSocket.MesagePump in SyncRequest, so I suppose 
creating timer will work, won't it?

-- 
Anton
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread francois . piette
 how can I add timeouts without messageloops and threads and so on?

You need a messageloop to make ICS component work anyway.
Every GUI application has a messageloop.
If you build a console mode application, you have to create one yourself. 
See various console mode ICS démos. Simple...

--
francois.pie...@overbyte.be
The author of the freeware multi-tier middleware MidWare
The author of the freeware Internet Component Suite (ICS)
http://www.overbyte.be--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be

Re: [twsocket] Timeout in sync operations

2009-09-16 Thread francois . piette

 And second question, more global: Isn't it worth to add good and native
 timeout functionality to TWSocket so that thousands of developers
 hadn't invent a bicycle again and again? I know about TIcsTimer
 and so on, but it is not 'out of the box'.

This is underway. but designing a general purpose timeout is not so simple 
because you potentially have thousands of components (server side). You 
can't create a TTimer for each one or you'll run out Windows resources very 
quickly (bad scalability).

Currently you should implement a timeout yourself. Using a single TTimer and 
carefully crafted code will do the job. Basically you need a list of all 
your timeout \jobs\ and have the timer event scan the list a do whatever 
appropriate is to evaluation and handle the timeout.
--
francois.pie...@overbyte.be
The author of the freeware multi-tier middleware MidWare
The author of the freeware Internet Component Suite (ICS)
http://www.overbyte.be
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Anton Sviridov
If you build a console mode application, you have to create one yourself. 
See various console mode ICS d?mos. Simple...
Yes, I created TIcsTimer with HttpCli.CtrlSocket as Owner and implemented 
Start/StopTimer methods. Seems working, although I don't see yet how can I 
control the main function flow from OnTimer procedure (i.e., how to tell 
CtrlSocket that is's timeout).

This is underway. but designing a general purpose timeout is not so simple 
because you potentially have thousands of components (server side). You 
can't create a TTimer for each one or you'll run out Windows resources very 
quickly (bad scalability).
No problem, you could implement only one timer in the very root of socket (as 
you did with socket's window handle, AFAIR).

Currently you should implement a timeout yourself. Using a single TTimer and 
carefully crafted code will do the job
Yeah, and every developer should do the same, so we are inventing the bicycle 
again. Maybe then some general TTimeoutOperationsSeeker? Anyway, at least for 
connection stage timeouts should be implemented, imho.

-- 
Anton
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Arno Garrels
Anton Sviridov wrote:
 If you build a console mode application, you have to create one
 yourself. See various console mode ICS d?mos. Simple...
 Yes, I created TIcsTimer with HttpCli.CtrlSocket as Owner and
 implemented Start/StopTimer methods. Seems working, although I don't
 see yet how can I control the main function flow from OnTimer
 procedure (i.e., how to tell CtrlSocket that is's timeout).

TWSocketCounter has been added to TWSocket in V7 with properties 
ConnectTick, ConnectDT, LastAliveTick, LastRecvTick and LastSendTick.
That makes it very easy to implement a custom timeout. 
Have a look at current THttpServer for an example.
   
 
 This is underway. but designing a general purpose timeout is not so
 simple because you potentially have thousands of components (server
 side). You can't create a TTimer for each one or you'll run out
 Windows resources very quickly (bad scalability).

 No problem, you could implement only one timer in the very root of
 socket (as you did with socket's window handle, AFAIR). 

If you have a better idea than TIcsThreadTimer, demo in the MiscDemos
folder, please let us know.   

--
Arno Garrels
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Fastream Technologies
I think Arno and Francois is right about not making a TTimer per Thttpcli as
default. It would overwhelm our reverse proxy with 10k connections each with
two sockets. However for simple needs, Anton is right about asking for a
solution which I think sould only be switched on by a define.

Just my two cents,

SZ
On Wed, Sep 16, 2009 at 8:01 PM, Arno Garrels arno.garr...@gmx.de wrote:

 Anton Sviridov wrote:
  If you build a console mode application, you have to create one
  yourself. See various console mode ICS d?mos. Simple...
  Yes, I created TIcsTimer with HttpCli.CtrlSocket as Owner and
  implemented Start/StopTimer methods. Seems working, although I don't
  see yet how can I control the main function flow from OnTimer
  procedure (i.e., how to tell CtrlSocket that is's timeout).

 TWSocketCounter has been added to TWSocket in V7 with properties
 ConnectTick, ConnectDT, LastAliveTick, LastRecvTick and LastSendTick.
 That makes it very easy to implement a custom timeout.
 Have a look at current THttpServer for an example.

 
  This is underway. but designing a general purpose timeout is not so
  simple because you potentially have thousands of components (server
  side). You can't create a TTimer for each one or you'll run out
  Windows resources very quickly (bad scalability).

  No problem, you could implement only one timer in the very root of
  socket (as you did with socket's window handle, AFAIR).

 If you have a better idea than TIcsThreadTimer, demo in the MiscDemos
 folder, please let us know.

 --
 Arno Garrels

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Arno Garrels
Anton Sviridov wrote:
 Hello!
 I'm writing simple console downloader and decided to use sync
 operations in order to simplify structure. But I've found that sync
 requests (HttpCli) do not use any timeout, so they may last forever
 in case of error. 

That's IMHO a bug.

--
Arno Garrels
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Timeout in sync operations

2009-09-16 Thread Angus Robertson - Magenta Systems Ltd
  I'm writing simple console downloader and decided to use sync
  operations in order to simplify structure. But I've found that 
  sync requests (HttpCli) do not use any timeout, so they may last 
  forever in case of error. 
 
 That's IMHO a bug.

Indeed, poor implementation in THttpCli with no timeout.  

But most components do have a timeout for sync methods, for instance
TSyncSmtpCli and TFtpClient.  

Angus

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be