Re: [twsocket] Showstopped bug in ICS

2011-07-05 Thread Wilfried Mestdagh
Hi Eric,

 I know, but if you still call OnDataAvailable with current socket
 options,
 for the ones that has not set wsoNoReceiveLoop (By default) the
 behaviour
 will still be the same right... for the ones that has set
 wsoNoReceiveLoop
 (like me) OnDataAvailable will still be called, and if I can afford to
 lose
 the packets 

But it will break existing code that depents on this behaviour if
wsoNoReceiveLoop is set. But you can easely derive from TWSocket and
override do_fd_close method. Then it will behave good for your
specifications and you will never have to change code again after updates.

-- 
mvg, Wilfried
http://www.mestdagh.biz
http://www.comfortsoftware.be
http://www.expertsoftware.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-05 Thread Éric Fleming Bonilha

Hi Wilfried


But it will break existing code that depents on this behaviour if
wsoNoReceiveLoop is set. But you can easely derive from TWSocket and
override do_fd_close method. Then it will behave good for your
specifications and you will never have to change code again after updates.


I see your point, I will override the do_fd_close method to fit my needs


Thanks 


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-04 Thread Éric Fleming Bonilha



Never done somthing like that, I guess you have to ensure that data is
processed in the right order with such a design, as with Windows IOCP
sockets as well.


No problem here, I have a worker thread system, the data will always be 
processed in right order



Yes, I understand, but there's a reason for doing this since it
ensures that OnSessionClosed does not fire until winsock buffer
is empty.


I see the point, but that would not have to be something that the user 
application should handle? Why does your library has to ensure that? What 
would harm if you call OnDataAvailable from FD_CLOSE using the current 
socket options (in my canse including wsoNoReceiveLoop)? I believe that the 
one that is using ICS should handle that, because ICS is doing its part, it 
is calling OnDataAvailable anyway, than, it is up to the user to read or not 
to read the socket



That may be a workaround in your special case if it doesn't matter
to lose same packets, however that isn't a general solution.
We have the same problem with SSL, where received encrypted data
has to be decrypted first before SessionClosed may be triggered,
the workaround is a derived component that overides Do_FD_Close.


I know, but if you still call OnDataAvailable with current socket options, 
for the ones that has not set wsoNoReceiveLoop (By default) the behaviour 
will still be the same right... for the ones that has set wsoNoReceiveLoop 
(like me) OnDataAvailable will still be called, and if I can afford to lose 
the packets I just don´t process the data, this will depend on the ones that 
are designing an application using ICS. In my workaround I´m still calling 
OnDataAvailable, that wasn´t change, but it is called with wsoNoReceiveLoop, 
so, it is up to my application to handle or not to handle the data, if my 
application does not handle it, it will lose packets (which in some cases 
are ok)


Eric 


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-02 Thread Arno Garrels
Éric Fleming Bonilha wrote:
 Arno
 
 I use multiple threads to read the data from sockets, I actually have
 a thread pool that process data.

Never done somthing like that, I guess you have to ensure that data is
processed in the right order with such a design, as with Windows IOCP
sockets as well.

 
 This is my current code for OnDataAvailable from sockets:
 
 procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender:
 TObject; ErrCode: Word);
 begin
 
   // We can´t proceed on errors
   if ErrCode  0 then
 Exit;
 
   // Start the work
   ScheduleAction(saReceiveData, 0);
 
 end;
 
 So, basically, it does not read the socket, but it schedules an
 action that will be processed by my thread pool system, a thread will
 get the scheduled action and will effectivelly read the socket.
 
 The problem is that Do_FD_CLOSE will call my
 HandleOnSocketDataAvailable routine, and since that does not read the
 data from socket, the library will enter in an infinite loop, locking
 up my whole system 

[..] 

 Hope you understand the problem I´m facing.

Yes, I understand, but there's a reason for doing this since it 
ensures that OnSessionClosed does not fire until winsock buffer
is empty. 

 
 To temporarily fix the issue I changed my ICS code to:
 
 if (FState  wsConnecting) and (FHSocket  INVALID_SOCKET) then
 begin { Check if we have something arrived, if yes, process it   
 } { DLR, since we are closing MAKE SURE WE LOOP in the receive }
 { function to get ALL remaining data }
 ASyncReceive(0, FComponentOptions);
 
 That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me
 not to process the data, because if it is closing the socket I can
 lose the remaining data anyway, this is video data

That may be a workaround in your special case if it doesn't matter
to lose same packets, however that isn't a general solution.
We have the same problem with SSL, where received encrypted data
has to be decrypted first before SessionClosed may be triggered,
the workaround is a derived component that overides Do_FD_Close.

-- 
Arno Garrels


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-02 Thread Wilfried Mestdagh
 I know this is by design, but I believe that is why you have added
 wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable...

No this is addes to get out of a closed loop in case of users had slow code
to process fast data.

 software wasn´t being able to process over 150mbits of incomming data
 because the data from all sockets were being read from a single thread.

It is easy to handel that amout of data in 1 thread. If processing is slow
(for example because it needs slow queries of database) then pass it to a
worker thread.

-- 
mvg, Wilfried
http://www.mestdagh.biz
http://www.comfortsoftware.be
http://www.expertsoftware.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-01 Thread Wilfried Mestdagh
Hi,

 it is expecting that my
 event handler DO receive the data (But it doesn´t) and then, it locks
 up on a loop!

You have to receive the available data in the OnDataAvailable event handler.
This is by design. If you want to process the data in another thread then
the most obvious is that you receive the data in the event handler and then
pass it to the other thread.

-- 
mvg, Wilfried
http://www.mestdagh.biz
http://www.comfortsoftware.be
http://www.expertsoftware.be

--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-01 Thread Éric Fleming Bonilha
I know this is by design, but I believe that is why you have added 
wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable...


By reading the socket using OnDataAvailable event I reached a bottleneck, my 
software wasn´t being able to process over 150mbits of incomming data 
because the data from all sockets were being read from a single thread.


So, when I receive the OnDataAvailable event, I trigger a Processing 
Thread to read the socket, by doing that I have increased significantly the 
amount of data I can process and in my tests, my application is now able to 
handle over 500mbits of incomming data without any suffering


You have to receive the available data in the OnDataAvailable event 
handler.

This is by design. If you want to process the data in another thread then
the most obvious is that you receive the data in the event handler and 
then

pass it to the other thread.


--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-01 Thread Arno Garrels
Éric Fleming Bonilha wrote:
 I know this is by design, but I believe that is why you have added
 wsoNoReceiveLoop so we don´t have to receive data in
 OnDataAvailable... 
 
 By reading the socket using OnDataAvailable event I reached a
 bottleneck, my software wasn´t being able to process over 150mbits of
 incomming data because the data from all sockets were being read from
 a single thread.
 
 So, when I receive the OnDataAvailable event, I trigger a Processing
 Thread to read the socket, by doing that I have increased
 significantly the amount of data I can process and in my tests, my
 application is now able to handle over 500mbits of incomming data
 without any suffering 

Do you use multiple threads or just one? If the latter you could simply
create and destroy the ICS object in TThread's Execute method.
If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set,
that cannot be much data after Do_FD_CLOSE has been called.

-- 
Arno Garrels



   
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-01 Thread Arno Garrels
Arno Garrels wrote:
 Éric Fleming Bonilha wrote:
 I know this is by design, but I believe that is why you have added
 wsoNoReceiveLoop so we don´t have to receive data in
 OnDataAvailable...
 
 By reading the socket using OnDataAvailable event I reached a
 bottleneck, my software wasn´t being able to process over 150mbits of
 incomming data because the data from all sockets were being read from
 a single thread.
 
 So, when I receive the OnDataAvailable event, I trigger a Processing
 Thread to read the socket, by doing that I have increased
 significantly the amount of data I can process and in my tests, my
 application is now able to handle over 500mbits of incomming data
 without any suffering
 
 Do you use multiple threads or just one? If the latter you could
 simply create and destroy the ICS object in TThread's Execute method.
 If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not
 set, that cannot be much data after Do_FD_CLOSE has been called.

Sorry I meant when wsoNoReceiveLoop _is set.




--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


Re: [twsocket] Showstopped bug in ICS

2011-07-01 Thread Éric Fleming Bonilha

Arno

I use multiple threads to read the data from sockets, I actually have a 
thread pool that process data.


This is my current code for OnDataAvailable from sockets:

procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender: TObject; 
ErrCode: Word);

begin

 // We can´t proceed on errors
 if ErrCode  0 then
   Exit;

 // Start the work
 ScheduleAction(saReceiveData, 0);

end;

So, basically, it does not read the socket, but it schedules an action that 
will be processed by my thread pool system, a thread will get the scheduled 
action and will effectivelly read the socket.


The problem is that Do_FD_CLOSE will call my HandleOnSocketDataAvailable 
routine, and since that does not read the data from socket, the library will 
enter in an infinite loop, locking up my whole system


This is my worker thread execution routine (This routine is called by a 
thread from the pool), I just left the relevant parts of the code here:


procedure TTCPSocketThread.Work;
var
 Action: TSocketActionRecord;
begin

 // Get the action to execute
 Action := GetScheduledAction;

 // Get a scheduled action and process it
 case Action.Action of

   ..
   ..

   // Action to receive data
   saReceiveData: TriggerOnSocketDataAvailable(0);

 end;

end;

My class TTCPSocketThread is basically an encapsulation of your socket, but 
then OnSocketDataAvailable is triggered it will be triggered by a thread 
from thread pool, this is required for my application since all the 
processing done outside the class on the handler of the data should be 
processed on threading and I found this was the best way to avoid many 
buffering techniques


Hope you understand the problem I´m facing.

To temporarily fix the issue I changed my ICS code to:

if (FState  wsConnecting) and (FHSocket  INVALID_SOCKET) then begin
   { Check if we have something arrived, if yes, process it }
   { DLR, since we are closing MAKE SURE WE LOOP in the receive }
   { function to get ALL remaining data }
   ASyncReceive(0, FComponentOptions);

That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me not to 
process the data, because if it is closing the socket I can lose the 
remaining data anyway, this is video data




Do you use multiple threads or just one? If the latter you could simply
create and destroy the ICS object in TThread's Execute method.
If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set,
that cannot be much data after Do_FD_CLOSE has been called.



--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be


[twsocket] Showstopped bug in ICS

2011-06-30 Thread Éric Fleming Bonilha
Hi

I believe I found a showstopper bug in ICS that lead my software to crash in 
many customers.

The problem is with wsoNoReceiveLoop option from socket!

In my program, I have to use wsoNoReceiveLoop in order to better process the 
data, I don´t want to process the data from the thread that fires the 
OnDataAvailable event, instead, I want to read the data from another thread... 
Thats fine... but.. I found this:

procedure TCustomWSocket.Do_FD_CLOSE(var msg: TMessage);
begin
{ In some strange situations I found that we receive a FD_CLOSE  }
{ during the connection phase, breaking the connection early !   }
{ This occurs for example after a failed FTP transfert Probably  }
{ something related to bugged winsock. Doesn't hurt with good}
{ winsock. So let the code there !   }
if (FState  wsConnecting) and (FHSocket  INVALID_SOCKET) then begin
{ Check if we have something arrived, if yes, process it }
{ DLR, since we are closing MAKE SURE WE LOOP in the receive }
{ function to get ALL remaining data }
ASyncReceive(0, RemoveOption(FComponentOptions, wsoNoReceiveLoop));

if not FCloseInvoked then begin
FCloseInvoked := TRUE;
TriggerSessionClosed(HiWord(msg.LParam));
end;

if FState  wsClosed then
Close;
end;
end;

On routine Do_FD_CLOSE, the ASyncReceive (Which triggers my OnDataAvailable 
event handler) can be called WITHOUT the wsoNoReceiveLoop.. guess what is 
happening? My OnDataAvailable event handler DOES NOT process the data from the 
socket, and since ASyncReceive was called with wsoNoReceiveLoop it is expecting 
that my event handler DO receive the data (But it doesn´t) and then, it locks 
up on a loop!

What can I do??

Eric
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be