Re: [twsocket] complex problem, probably not ICS related
> to my problem I found something out: if I disable processing of these > commands up to 100.000 and more are stored in the queue, so the sending > and receiving of them via ICS works well. :-) > The receiving uses a tight loop to extract all commands out of the > ringbuffer and stores them in the queue. After storing one command the > processing thread will be informed via PostThreadMessage, but obviously > it doesn't come to work, maybe that other loop is too tight and occurs > all the time so the other thread never gets time to do anything real. Your thread must have a valid working message pump to retrieve messages posted by PostThreadMessage. You your thread is suspended or sleeping or busy in a loop, the message pump is not working anymore and no message is processed. > My idea is now to slow down this tight loop just a tiny little bit so > that the other thread can do his work. I don't clearly understand where this tight loop is. Is it in the main thread, in the worker thread or somewhere else ? I any wase, you should never has loops. You must think asynchrnous with ICS and think asynchronous with your thread. The main thread and your worker thread should be blocked in their respective message loop (GetMessage or MsgWaitForMultipleObject). > I could do it via sleep Sleep is not good, at least at ICS side. Sleep will stop the thread. No message will be handled, no event will be processed. > or I could stop TWSocket's receiving for a short time. You can do that using TWSocet.Pause/Resume. For example if you detect your message list has too much messages not processed yet. This could occur when you have a fast network compared to the processing time you need. If you have complete asynchronous receive, the network may overflow your system if you don't stop receiving when you've got already too much data not processed yet. > Or I could change my protocol so that the next command may only > be sent after the receiver acknowledged the prior command. Most protocols are implemented that way. This has the good side effect of having the sender know when the receiver detected some problem in the data. Without ACK, the sender doesn't know his data has been processed properly. Implementing ACK may slow down overall operation. If you want to have a high performance thruput, you may want to implement a windowed protocol, that is allow the sender to send many commands without having received the ACK. The sender will stop sending more commands if no ACK is received for let's say 10 commands back (the number of pending commands is the window size). This kind of "windowed protocol" is not trivial to implement but offer high performance where thruput is high but round-trip is slow. > a) what would be better/best? It's up to you to know. I gave you the ideas. > b) what is the minimum time sleep can sleep (Win2K/XP) Sleep(0) will just give back the remaining of the time slice a thread has. He will get CPU back as soon as possible given his priority and all other threads running in the system. A time slice is something like 20mS. > c) how to pause and resume TWSocket and can there data be lost? There are two method Pause and Resume. No data can be lost, but when you call resume, you don't get any notification before the next packet, so maybe data is waiting in the buffer. You could call Receive to get it (returns <0 if no data available, if remote has closed or nb of bytes if any data was read). btw: I took the time to write this long answer because you are actively writing in the wiki. A kind of compensation for your work. Thanks. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] complex problem, probably not ICS related
Hello, to my problem I found something out: if I disable processing of these commands up to 100.000 and more are stored in the queue, so the sending and receiving of them via ICS works well. The receiving uses a tight loop to extract all commands out of the ringbuffer and stores them in the queue. After storing one command the processing thread will be informed via PostThreadMessage, but obviously it doesn't come to work, maybe that other loop is too tight and occurs all the time so the other thread never gets time to do anything real. My idea is now to slow down this tight loop just a tiny little bit so that the other thread can do his work. I could do it via sleep or I could stop TWSocket's receiving for a short time. Or I could change my protocol so that the next command may only be sent after the receiver acknowledged the prior command. If some fool would'nt keep at this and still send in a tight loop he could overrun the server, but since all senders will be programmed by me this won't be a problem (unless somebody tries to "misuse" the connection). a) what would be better/best? b) what is the minimum time sleep can sleep (Win2K/XP) c) how to pause and resume TWSocket and can there data be lost? Greetings Markus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] complex problem, probably not ICS related
Try disabling your thread sleeping (the thread that reads from the StringList). Does your stringlist get all the the requests? i.e. is TWSocket receiving them and your code parsing and putting them in the list correctly? Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Markus Humm Sent: 14 September 2006 21:54 To: ICS support mailing Subject: [twsocket] complex problem, probably not ICS related Hello, sorry, this will be just a tiny bit longer... ;-) I'm currently building some large application where several parts will act together. Two of them are connected via TCP and TWSocket. One is client and one server (okay there is TWSocketServer, but since there's always a 1:1 relation between them and I don't have time to learn [and wikify] all ICS components right now...). The client sends requests to the server and the server will send back the replies in the same format. This format is as follows: 1 integer defining the length of my packet 1 integer some ID I need x byte the other needed data, can be any size but mostly < 100 bytes 1 word CRC so that it doesn't get too easy to spoof my connection. The receiver puts incomming bytes into a ringbuffer with a greater size than the buffer where TWSocket.receive puts the data in in the first place. Then it checks whether there are packets of my format in the ringpuffer and as long as there is one, one is fetched and given to a callback in my application. There the syntax of the x byte is checked and it is added to a TStringList. Reading and writing to the TStringList is protected by the same critical section. On the other side of the TStringList is a thread which will fetach one entry and process it (will take around 50ms), but requests can come in as fast as possible, thus this TStringList. If no requests are left in the TStringList the threads gets sleepy (GetMessage) until something is put into the TStringList again, which also will inform the Thread via PostThreadMessage. Replies generated from processing will be in the same packet format and will be stored in another TStringList where another thread will fetch one after one and send them back to the requester via the same TWSocket. For the reply TStringList another critical section is used to protect it. I have now a simple test client where I can send packets manually via button or 1000 in one go via a for loop. The test app. displays the replies in a memo. If I send some requests manually, all is fine, but if I send them in the loop (no sleep between the iterations) the whole thing gets stuck somewhere. At least no answers get back and it seems that these requests are added to the request TStringList (I can see the count of it) but something prevents further processing. This is a real block for me now. Now I want to rule out that it has something todo with the TWSocket and or rinbuffer implementation. Do critical sections work in the same thread or only for different threads? Greetings Markus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] complex problem, probably not ICS related
> Do critical sections work in the same > thread or only for different threads? Critical section work only for different threads. > The receiver puts incomming bytes into a ringbuffer with a greater size > than the buffer where TWSocket.receive puts the data in in the first > place. Then it checks whether there are packets of my format in the > ringpuffer and as long as there is one, one is fetched and given to a > callback in my application. > I send them in the loop (no sleep between the iterations) the whole > thing gets stuck somewhere. When you send data using a for loop you will receive more than one of your packets into a single call to receive unless you first receive the length (which could be fragmented) and then call receive to receive at most the length of your packet (You may receive less !). You must really pay attention to your code to receive your data. Always think you'll receive less than what you would like, or more ! If you change your packet format slightly, you life may be easier: insted of length and data, send data and delimiter. You'll then be able to use TWSocket line mode. Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] Author of ICS (Internet Component Suite, freeware) Author of MidWare (Multi-tier framework, freeware) http://www.overbyte.be - Original Message - From: "Markus Humm" <[EMAIL PROTECTED]> To: "ICS support mailing" Sent: Thursday, September 14, 2006 10:53 PM Subject: [twsocket] complex problem, probably not ICS related > Hello, > > sorry, this will be just a tiny bit longer... ;-) > > I'm currently building some large application where several parts will > act together. Two of them are connected via TCP and TWSocket. One is > client and one server (okay there is TWSocketServer, but since there's > always a 1:1 relation between them and I don't have time to learn [and > wikify] all ICS components right now...). > > The client sends requests to the server and the server will send back > the replies in the same format. This format is as follows: > > 1 integer defining the length of my packet > 1 integer some ID I need > x byte the other needed data, can be any size but mostly < 100 bytes > 1 word CRC so that it doesn't get too easy to spoof my connection. > > The receiver puts incomming bytes into a ringbuffer with a greater size > than the buffer where TWSocket.receive puts the data in in the first > place. Then it checks whether there are packets of my format in the > ringpuffer and as long as there is one, one is fetched and given to a > callback in my application. There the syntax of the x byte is checked > and it is added to a TStringList. Reading and writing to the TStringList > is protected by the same critical section. > > On the other side of the TStringList is a thread which will fetach one > entry and process it (will take around 50ms), but requests can come in > as fast as possible, thus this TStringList. If no requests are left in > the TStringList the threads gets sleepy (GetMessage) until something is > put into the TStringList again, which also will inform the Thread via > PostThreadMessage. > > Replies generated from processing will be in the same packet format and > will be stored in another TStringList where another thread will fetch > one after one and send them back to the requester via the same TWSocket. > For the reply TStringList another critical section is used to protect it. > > I have now a simple test client where I can send packets manually via > button or 1000 in one go via a for loop. The test app. displays the > replies in a memo. If I send some requests manually, all is fine, but if > I send them in the loop (no sleep between the iterations) the whole > thing gets stuck somewhere. At least no answers get back and it seems > that these requests are added to the request TStringList (I can see the > count of it) but something prevents further processing. This is a real > block for me now. > > Now I want to rule out that it has something todo with the TWSocket and > or rinbuffer implementation. Do critical sections work in the same > thread or only for different threads? > > Greetings > > Markus > > -- > To unsubscribe or change your settings for TWSocket mailing list > please goto http://www.elists.org/mailman/listinfo/twsocket > Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] complex problem, probably not ICS related
Hello, sorry, this will be just a tiny bit longer... ;-) I'm currently building some large application where several parts will act together. Two of them are connected via TCP and TWSocket. One is client and one server (okay there is TWSocketServer, but since there's always a 1:1 relation between them and I don't have time to learn [and wikify] all ICS components right now...). The client sends requests to the server and the server will send back the replies in the same format. This format is as follows: 1 integer defining the length of my packet 1 integer some ID I need x byte the other needed data, can be any size but mostly < 100 bytes 1 word CRC so that it doesn't get too easy to spoof my connection. The receiver puts incomming bytes into a ringbuffer with a greater size than the buffer where TWSocket.receive puts the data in in the first place. Then it checks whether there are packets of my format in the ringpuffer and as long as there is one, one is fetched and given to a callback in my application. There the syntax of the x byte is checked and it is added to a TStringList. Reading and writing to the TStringList is protected by the same critical section. On the other side of the TStringList is a thread which will fetach one entry and process it (will take around 50ms), but requests can come in as fast as possible, thus this TStringList. If no requests are left in the TStringList the threads gets sleepy (GetMessage) until something is put into the TStringList again, which also will inform the Thread via PostThreadMessage. Replies generated from processing will be in the same packet format and will be stored in another TStringList where another thread will fetch one after one and send them back to the requester via the same TWSocket. For the reply TStringList another critical section is used to protect it. I have now a simple test client where I can send packets manually via button or 1000 in one go via a for loop. The test app. displays the replies in a memo. If I send some requests manually, all is fine, but if I send them in the loop (no sleep between the iterations) the whole thing gets stuck somewhere. At least no answers get back and it seems that these requests are added to the request TStringList (I can see the count of it) but something prevents further processing. This is a real block for me now. Now I want to rule out that it has something todo with the TWSocket and or rinbuffer implementation. Do critical sections work in the same thread or only for different threads? Greetings Markus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
