Re: [twsocket] Showstopped bug in ICS
Hi Eric, I know, but if you still call OnDataAvailable with current socket options, for the ones that has not set wsoNoReceiveLoop (By default) the behaviour will still be the same right... for the ones that has set wsoNoReceiveLoop (like me) OnDataAvailable will still be called, and if I can afford to lose the packets But it will break existing code that depents on this behaviour if wsoNoReceiveLoop is set. But you can easely derive from TWSocket and override do_fd_close method. Then it will behave good for your specifications and you will never have to change code again after updates. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Never done somthing like that, I guess you have to ensure that data is processed in the right order with such a design, as with Windows IOCP sockets as well. No problem here, I have a worker thread system, the data will always be processed in right order Yes, I understand, but there's a reason for doing this since it ensures that OnSessionClosed does not fire until winsock buffer is empty. I see the point, but that would not have to be something that the user application should handle? Why does your library has to ensure that? What would harm if you call OnDataAvailable from FD_CLOSE using the current socket options (in my canse including wsoNoReceiveLoop)? I believe that the one that is using ICS should handle that, because ICS is doing its part, it is calling OnDataAvailable anyway, than, it is up to the user to read or not to read the socket That may be a workaround in your special case if it doesn't matter to lose same packets, however that isn't a general solution. We have the same problem with SSL, where received encrypted data has to be decrypted first before SessionClosed may be triggered, the workaround is a derived component that overides Do_FD_Close. I know, but if you still call OnDataAvailable with current socket options, for the ones that has not set wsoNoReceiveLoop (By default) the behaviour will still be the same right... for the ones that has set wsoNoReceiveLoop (like me) OnDataAvailable will still be called, and if I can afford to lose the packets I just don´t process the data, this will depend on the ones that are designing an application using ICS. In my workaround I´m still calling OnDataAvailable, that wasn´t change, but it is called with wsoNoReceiveLoop, so, it is up to my application to handle or not to handle the data, if my application does not handle it, it will lose packets (which in some cases are ok) Eric -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Éric Fleming Bonilha wrote:
Arno
I use multiple threads to read the data from sockets, I actually have
a thread pool that process data.
Never done somthing like that, I guess you have to ensure that data is
processed in the right order with such a design, as with Windows IOCP
sockets as well.
This is my current code for OnDataAvailable from sockets:
procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender:
TObject; ErrCode: Word);
begin
// We can´t proceed on errors
if ErrCode 0 then
Exit;
// Start the work
ScheduleAction(saReceiveData, 0);
end;
So, basically, it does not read the socket, but it schedules an
action that will be processed by my thread pool system, a thread will
get the scheduled action and will effectivelly read the socket.
The problem is that Do_FD_CLOSE will call my
HandleOnSocketDataAvailable routine, and since that does not read the
data from socket, the library will enter in an infinite loop, locking
up my whole system
[..]
Hope you understand the problem I´m facing.
Yes, I understand, but there's a reason for doing this since it
ensures that OnSessionClosed does not fire until winsock buffer
is empty.
To temporarily fix the issue I changed my ICS code to:
if (FState wsConnecting) and (FHSocket INVALID_SOCKET) then
begin { Check if we have something arrived, if yes, process it
} { DLR, since we are closing MAKE SURE WE LOOP in the receive }
{ function to get ALL remaining data }
ASyncReceive(0, FComponentOptions);
That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me
not to process the data, because if it is closing the socket I can
lose the remaining data anyway, this is video data
That may be a workaround in your special case if it doesn't matter
to lose same packets, however that isn't a general solution.
We have the same problem with SSL, where received encrypted data
has to be decrypted first before SessionClosed may be triggered,
the workaround is a derived component that overides Do_FD_Close.
--
Arno Garrels
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
I know this is by design, but I believe that is why you have added wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... No this is addes to get out of a closed loop in case of users had slow code to process fast data. software wasn´t being able to process over 150mbits of incomming data because the data from all sockets were being read from a single thread. It is easy to handel that amout of data in 1 thread. If processing is slow (for example because it needs slow queries of database) then pass it to a worker thread. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Hi, it is expecting that my event handler DO receive the data (But it doesn´t) and then, it locks up on a loop! You have to receive the available data in the OnDataAvailable event handler. This is by design. If you want to process the data in another thread then the most obvious is that you receive the data in the event handler and then pass it to the other thread. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
I know this is by design, but I believe that is why you have added wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... By reading the socket using OnDataAvailable event I reached a bottleneck, my software wasn´t being able to process over 150mbits of incomming data because the data from all sockets were being read from a single thread. So, when I receive the OnDataAvailable event, I trigger a Processing Thread to read the socket, by doing that I have increased significantly the amount of data I can process and in my tests, my application is now able to handle over 500mbits of incomming data without any suffering You have to receive the available data in the OnDataAvailable event handler. This is by design. If you want to process the data in another thread then the most obvious is that you receive the data in the event handler and then pass it to the other thread. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Éric Fleming Bonilha wrote: I know this is by design, but I believe that is why you have added wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... By reading the socket using OnDataAvailable event I reached a bottleneck, my software wasn´t being able to process over 150mbits of incomming data because the data from all sockets were being read from a single thread. So, when I receive the OnDataAvailable event, I trigger a Processing Thread to read the socket, by doing that I have increased significantly the amount of data I can process and in my tests, my application is now able to handle over 500mbits of incomming data without any suffering Do you use multiple threads or just one? If the latter you could simply create and destroy the ICS object in TThread's Execute method. If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set, that cannot be much data after Do_FD_CLOSE has been called. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Arno Garrels wrote: Éric Fleming Bonilha wrote: I know this is by design, but I believe that is why you have added wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... By reading the socket using OnDataAvailable event I reached a bottleneck, my software wasn´t being able to process over 150mbits of incomming data because the data from all sockets were being read from a single thread. So, when I receive the OnDataAvailable event, I trigger a Processing Thread to read the socket, by doing that I have increased significantly the amount of data I can process and in my tests, my application is now able to handle over 500mbits of incomming data without any suffering Do you use multiple threads or just one? If the latter you could simply create and destroy the ICS object in TThread's Execute method. If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set, that cannot be much data after Do_FD_CLOSE has been called. Sorry I meant when wsoNoReceiveLoop _is set. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Arno
I use multiple threads to read the data from sockets, I actually have a
thread pool that process data.
This is my current code for OnDataAvailable from sockets:
procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender: TObject;
ErrCode: Word);
begin
// We can´t proceed on errors
if ErrCode 0 then
Exit;
// Start the work
ScheduleAction(saReceiveData, 0);
end;
So, basically, it does not read the socket, but it schedules an action that
will be processed by my thread pool system, a thread will get the scheduled
action and will effectivelly read the socket.
The problem is that Do_FD_CLOSE will call my HandleOnSocketDataAvailable
routine, and since that does not read the data from socket, the library will
enter in an infinite loop, locking up my whole system
This is my worker thread execution routine (This routine is called by a
thread from the pool), I just left the relevant parts of the code here:
procedure TTCPSocketThread.Work;
var
Action: TSocketActionRecord;
begin
// Get the action to execute
Action := GetScheduledAction;
// Get a scheduled action and process it
case Action.Action of
..
..
// Action to receive data
saReceiveData: TriggerOnSocketDataAvailable(0);
end;
end;
My class TTCPSocketThread is basically an encapsulation of your socket, but
then OnSocketDataAvailable is triggered it will be triggered by a thread
from thread pool, this is required for my application since all the
processing done outside the class on the handler of the data should be
processed on threading and I found this was the best way to avoid many
buffering techniques
Hope you understand the problem I´m facing.
To temporarily fix the issue I changed my ICS code to:
if (FState wsConnecting) and (FHSocket INVALID_SOCKET) then begin
{ Check if we have something arrived, if yes, process it }
{ DLR, since we are closing MAKE SURE WE LOOP in the receive }
{ function to get ALL remaining data }
ASyncReceive(0, FComponentOptions);
That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me not to
process the data, because if it is closing the socket I can lose the
remaining data anyway, this is video data
Do you use multiple threads or just one? If the latter you could simply
create and destroy the ICS object in TThread's Execute method.
If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set,
that cannot be much data after Do_FD_CLOSE has been called.
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
