Re: [twsocket] Showstopped bug in ICS
Hi Wilfried But it will break existing code that depents on this behaviour if wsoNoReceiveLoop is set. But you can easely derive from TWSocket and override do_fd_close method. Then it will behave good for your specifications and you will never have to change code again after updates. I see your point, I will override the do_fd_close method to fit my needs Thanks -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Hi Eric, > I know, but if you still call OnDataAvailable with current socket > options, > for the ones that has not set wsoNoReceiveLoop (By default) the > behaviour > will still be the same right... for the ones that has set > wsoNoReceiveLoop > (like me) OnDataAvailable will still be called, and if I can afford to > lose > the packets But it will break existing code that depents on this behaviour if wsoNoReceiveLoop is set. But you can easely derive from TWSocket and override do_fd_close method. Then it will behave good for your specifications and you will never have to change code again after updates. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Never done somthing like that, I guess you have to ensure that data is processed in the right order with such a design, as with Windows IOCP sockets as well. No problem here, I have a worker thread system, the data will always be processed in right order Yes, I understand, but there's a reason for doing this since it ensures that OnSessionClosed does not fire until winsock buffer is empty. I see the point, but that would not have to be something that the user application should handle? Why does your library has to ensure that? What would harm if you call OnDataAvailable from FD_CLOSE using the current socket options (in my canse including wsoNoReceiveLoop)? I believe that the one that is using ICS should handle that, because ICS is doing its part, it is calling OnDataAvailable anyway, than, it is up to the user to read or not to read the socket That may be a workaround in your special case if it doesn't matter to lose same packets, however that isn't a general solution. We have the same problem with SSL, where received encrypted data has to be decrypted first before SessionClosed may be triggered, the workaround is a derived component that overides Do_FD_Close. I know, but if you still call OnDataAvailable with current socket options, for the ones that has not set wsoNoReceiveLoop (By default) the behaviour will still be the same right... for the ones that has set wsoNoReceiveLoop (like me) OnDataAvailable will still be called, and if I can afford to lose the packets I just don´t process the data, this will depend on the ones that are designing an application using ICS. In my workaround I´m still calling OnDataAvailable, that wasn´t change, but it is called with wsoNoReceiveLoop, so, it is up to my application to handle or not to handle the data, if my application does not handle it, it will lose packets (which in some cases are ok) Eric -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
> I know this is by design, but I believe that is why you have added > wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... No this is addes to get out of a closed loop in case of users had slow code to process fast data. > software wasn´t being able to process over 150mbits of incomming data > because the data from all sockets were being read from a single thread. It is easy to handel that amout of data in 1 thread. If processing is slow (for example because it needs slow queries of database) then pass it to a worker thread. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Éric Fleming Bonilha wrote: > Arno > > I use multiple threads to read the data from sockets, I actually have > a thread pool that process data. Never done somthing like that, I guess you have to ensure that data is processed in the right order with such a design, as with Windows IOCP sockets as well. > > This is my current code for OnDataAvailable from sockets: > > procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender: > TObject; ErrCode: Word); > begin > > // We can´t proceed on errors > if ErrCode <> 0 then > Exit; > > // Start the work > ScheduleAction(saReceiveData, 0); > > end; > > So, basically, it does not read the socket, but it schedules an > action that will be processed by my thread pool system, a thread will > get the scheduled action and will effectivelly read the socket. > > The problem is that Do_FD_CLOSE will call my > HandleOnSocketDataAvailable routine, and since that does not read the > data from socket, the library will enter in an infinite loop, locking > up my whole system [..] > Hope you understand the problem I´m facing. Yes, I understand, but there's a reason for doing this since it ensures that OnSessionClosed does not fire until winsock buffer is empty. > > To temporarily fix the issue I changed my ICS code to: > > if (FState <> wsConnecting) and (FHSocket <> INVALID_SOCKET) then > begin { Check if we have something arrived, if yes, process it > } { DLR, since we are closing MAKE SURE WE LOOP in the receive } > { function to get ALL remaining data } > ASyncReceive(0, FComponentOptions); > > That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me > not to process the data, because if it is closing the socket I can > lose the remaining data anyway, this is video data That may be a workaround in your special case if it doesn't matter to lose same packets, however that isn't a general solution. We have the same problem with SSL, where received encrypted data has to be decrypted first before SessionClosed may be triggered, the workaround is a derived component that overides Do_FD_Close. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Arno I use multiple threads to read the data from sockets, I actually have a thread pool that process data. This is my current code for OnDataAvailable from sockets: procedure TTCPSocketThread.HandleOnSocketDataAvailable(Sender: TObject; ErrCode: Word); begin // We can´t proceed on errors if ErrCode <> 0 then Exit; // Start the work ScheduleAction(saReceiveData, 0); end; So, basically, it does not read the socket, but it schedules an action that will be processed by my thread pool system, a thread will get the scheduled action and will effectivelly read the socket. The problem is that Do_FD_CLOSE will call my HandleOnSocketDataAvailable routine, and since that does not read the data from socket, the library will enter in an infinite loop, locking up my whole system This is my worker thread execution routine (This routine is called by a thread from the pool), I just left the relevant parts of the code here: procedure TTCPSocketThread.Work; var Action: TSocketActionRecord; begin // Get the action to execute Action := GetScheduledAction; // Get a scheduled action and process it case Action.Action of .. .. // Action to receive data saReceiveData: TriggerOnSocketDataAvailable(0); end; end; My class TTCPSocketThread is basically an encapsulation of your socket, but then OnSocketDataAvailable is triggered it will be triggered by a thread from thread pool, this is required for my application since all the processing done outside the class on the handler of the data should be processed on threading and I found this was the best way to avoid many buffering techniques Hope you understand the problem I´m facing. To temporarily fix the issue I changed my ICS code to: if (FState <> wsConnecting) and (FHSocket <> INVALID_SOCKET) then begin { Check if we have something arrived, if yes, process it } { DLR, since we are closing MAKE SURE WE LOOP in the receive } { function to get ALL remaining data } ASyncReceive(0, FComponentOptions); That will call ASyncReceive with flag wsoNoReceiveLoop, allowing me not to process the data, because if it is closing the socket I can lose the remaining data anyway, this is video data Do you use multiple threads or just one? If the latter you could simply create and destroy the ICS object in TThread's Execute method. If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set, that cannot be much data after Do_FD_CLOSE has been called. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Arno Garrels wrote: > Éric Fleming Bonilha wrote: >> I know this is by design, but I believe that is why you have added >> wsoNoReceiveLoop so we don´t have to receive data in >> OnDataAvailable... >> >> By reading the socket using OnDataAvailable event I reached a >> bottleneck, my software wasn´t being able to process over 150mbits of >> incomming data because the data from all sockets were being read from >> a single thread. >> >> So, when I receive the OnDataAvailable event, I trigger a "Processing >> Thread" to read the socket, by doing that I have increased >> significantly the amount of data I can process and in my tests, my >> application is now able to handle over 500mbits of incomming data >> without any suffering > > Do you use multiple threads or just one? If the latter you could > simply create and destroy the ICS object in TThread's Execute method. > If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not > set, that cannot be much data after Do_FD_CLOSE has been called. Sorry I meant when wsoNoReceiveLoop _is set. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Éric Fleming Bonilha wrote: > I know this is by design, but I believe that is why you have added > wsoNoReceiveLoop so we don´t have to receive data in > OnDataAvailable... > > By reading the socket using OnDataAvailable event I reached a > bottleneck, my software wasn´t being able to process over 150mbits of > incomming data because the data from all sockets were being read from > a single thread. > > So, when I receive the OnDataAvailable event, I trigger a "Processing > Thread" to read the socket, by doing that I have increased > significantly the amount of data I can process and in my tests, my > application is now able to handle over 500mbits of incomming data > without any suffering Do you use multiple threads or just one? If the latter you could simply create and destroy the ICS object in TThread's Execute method. If the first, receive in OnDataAvailable if wsoNoReceiveLoop is not set, that cannot be much data after Do_FD_CLOSE has been called. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
I know this is by design, but I believe that is why you have added wsoNoReceiveLoop so we don´t have to receive data in OnDataAvailable... By reading the socket using OnDataAvailable event I reached a bottleneck, my software wasn´t being able to process over 150mbits of incomming data because the data from all sockets were being read from a single thread. So, when I receive the OnDataAvailable event, I trigger a "Processing Thread" to read the socket, by doing that I have increased significantly the amount of data I can process and in my tests, my application is now able to handle over 500mbits of incomming data without any suffering You have to receive the available data in the OnDataAvailable event handler. This is by design. If you want to process the data in another thread then the most obvious is that you receive the data in the event handler and then pass it to the other thread. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Showstopped bug in ICS
Hi, > it is expecting that my > event handler DO receive the data (But it doesn´t) and then, it locks > up on a loop! You have to receive the available data in the OnDataAvailable event handler. This is by design. If you want to process the data in another thread then the most obvious is that you receive the data in the event handler and then pass it to the other thread. -- mvg, Wilfried http://www.mestdagh.biz http://www.comfortsoftware.be http://www.expertsoftware.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Showstopped bug in ICS
Hi I believe I found a showstopper bug in ICS that lead my software to crash in many customers. The problem is with wsoNoReceiveLoop option from socket! In my program, I have to use wsoNoReceiveLoop in order to better process the data, I don´t want to process the data from the thread that fires the OnDataAvailable event, instead, I want to read the data from another thread... Thats fine... but.. I found this: procedure TCustomWSocket.Do_FD_CLOSE(var msg: TMessage); begin { In some strange situations I found that we receive a FD_CLOSE } { during the connection phase, breaking the connection early ! } { This occurs for example after a failed FTP transfert Probably } { something related to bugged winsock. Doesn't hurt with good} { winsock. So let the code there ! } if (FState <> wsConnecting) and (FHSocket <> INVALID_SOCKET) then begin { Check if we have something arrived, if yes, process it } { DLR, since we are closing MAKE SURE WE LOOP in the receive } { function to get ALL remaining data } ASyncReceive(0, RemoveOption(FComponentOptions, wsoNoReceiveLoop)); if not FCloseInvoked then begin FCloseInvoked := TRUE; TriggerSessionClosed(HiWord(msg.LParam)); end; if FState <> wsClosed then Close; end; end; On routine Do_FD_CLOSE, the ASyncReceive (Which triggers my OnDataAvailable event handler) can be called WITHOUT the wsoNoReceiveLoop.. guess what is happening? My OnDataAvailable event handler DOES NOT process the data from the socket, and since ASyncReceive was called with wsoNoReceiveLoop it is expecting that my event handler DO receive the data (But it doesn´t) and then, it locks up on a loop! What can I do?? Eric -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be