Re: [twsocket] Very strange problem with server and client software [thanks]...
Hello. Just wanted to say thanks for the great input over the weekend! I will consider the input. Francois, you are right, that particular part of the initial handshaking exchange probably is a flaw in the implementation, although it is a requirement in the protocol design. In the protocol design, it is not possible to prepare the CHANNEL before sending the packet, as modifying the channel parameters before the Client is ready will result in a garbled reception by the Client. The packet buffering to prepare the packet before re-syncing the channel works fine. That is the only place in the protocol design where that is a weakness. Every other packet exchange is basically non-modal and can occur in any random or unpredictable order. I guess I assumed that because I was not RECEIVING multi-threaded, or calling any message processing loops, or implementing any custom message pumps, that I wouldn't experience a sequencing issue of that nature. Obviously, I was wrong. :-) Thanks again. Hoby -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Very strange problem with server and client software [solution]...
Hello Hoby, If PutDataInSendBuffer and Send 0 at the end makes a difference then you probably have a problem in the receiver. Same reason as the reason it depent from machine to machine. You can try also with SocketSpy, if it also make a difference then you sure have to check over there. - You mention mulitthreaded = 'True. This means that the TWSocket is Created in Execute method of a thread, or Ataccehd to a tread. Did you do one of both ? - Are you sure never message pump is called from in OnDataAvailable method, or any other method of TWSocket ? --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Friday, July 28, 2006, 23:32, Hoby Smith wrote: Hello. Well, ultimately, here is my solution. This appears to work 99.9% of the time, maybe 100%. There was one failure, but not sure if it was a related problem... ;) Oddly, turning on the TWSocket MultiThreaded property tends to alleviate the majority of symptoms. As I understand, the only major difference when MultiThreaded is on is that the ClientSocket utilizes its own messaging interface? Right? As for the issue I mentioned in the last email I posted, I solved the sequencing issue this way. Previously, I had a global send packet routine, wrapped in a Critical Section for thread safety. I use a single Global send routine so that I can gather statistics in one place (total bytes sent, total packets sent, total session bytes, etc). Since the MultiThreaded message loop was picking up responses before I was prepared to receive them, I created an additional couple of global routines. One is a global post packet routine, which processes the packet, but uses PutDataInSendBuffer to prepare and que up the outbound traffic (framing, encryption, etc), instead of sending it immediately. Then, when I am prepared to handle any related responses, I call a new global Send Buffer procedure, that simply issues a 0 byte Send on the ClientSocket to begin sending the que'd data. I then changed the sequence order sensitive handshaking routines to use the PostPacket / SendBuffer routines instead. This approach has appeared to fix the issue sufficiently. I am perplexed as to the nature of the issue itself, but for now, this will suffice. The scenario of both programs (client and server) being on one system is never going to occur during a live production environment. Solving this issue is really only relevant for development, testing and demonstration purposes. None-the-less, I am concerned that the surfacing of this issue might reflect some deeper logic flaw in the system as a whole. Does this solution sound appropriate? It appears to work fine. And does the problem issue sound like it should occur at all? Or does it sound like I have done something really BAD? I still can't get over the fact that it works fine and never occurs on the proverbial my machine... :) Thanks... Hoby -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Very strange problem with server and client software[solution]...
As I understand, you where - before you fix - sending some data which will result is some response being received, but at the time you send data your code is not yet ready to process the response. When runnin on different computers, there was enough time between the time you send the data and the time you receive the response to setup your program to process the response. But when running on the same computer, you get the response very quickly and your program is not able to prepare to receive the response. This is probably caused by the task switch. When your program sent the data, and a task switch occur, the sender loose the CPU, the receiver receive it and has enough time to build and send his response during a single time slice he received. The the sending program has CPU back but has not yet prepared itself to receive data and you have a problem. None-the-less, I am concerned that the surfacing of this issue might reflect some deeper logic flaw in the system as a whole. If what I understood is correct, I think you have a flow in the design. You must ALWAYS be prepared to process a response before sending a request, even when your client and server programs run on different computers. Then sending program may always loose CPU just after having send his request and may receive it back only when the response has already been received by winsock. See from the sending program, the response is received instantly after the request has been sent. That is why the program has to be able to process the reply without any delay. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be - Original Message - From: Hoby Smith [EMAIL PROTECTED] To: twsocket@elists.org Sent: Friday, July 28, 2006 11:32 PM Subject: Re: [twsocket] Very strange problem with server and client software[solution]... Hello. Well, ultimately, here is my solution. This appears to work 99.9% of the time, maybe 100%. There was one failure, but not sure if it was a related problem... ;) Oddly, turning on the TWSocket MultiThreaded property tends to alleviate the majority of symptoms. As I understand, the only major difference when MultiThreaded is on is that the ClientSocket utilizes its own messaging interface? Right? As for the issue I mentioned in the last email I posted, I solved the sequencing issue this way. Previously, I had a global send packet routine, wrapped in a Critical Section for thread safety. I use a single Global send routine so that I can gather statistics in one place (total bytes sent, total packets sent, total session bytes, etc). Since the MultiThreaded message loop was picking up responses before I was prepared to receive them, I created an additional couple of global routines. One is a global post packet routine, which processes the packet, but uses PutDataInSendBuffer to prepare and que up the outbound traffic (framing, encryption, etc), instead of sending it immediately. Then, when I am prepared to handle any related responses, I call a new global Send Buffer procedure, that simply issues a 0 byte Send on the ClientSocket to begin sending the que'd data. I then changed the sequence order sensitive handshaking routines to use the PostPacket / SendBuffer routines instead. This approach has appeared to fix the issue sufficiently. I am perplexed as to the nature of the issue itself, but for now, this will suffice. The scenario of both programs (client and server) being on one system is never going to occur during a live production environment. Solving this issue is really only relevant for development, testing and demonstration purposes. None-the-less, I am concerned that the surfacing of this issue might reflect some deeper logic flaw in the system as a whole. Does this solution sound appropriate? It appears to work fine. And does the problem issue sound like it should occur at all? Or does it sound like I have done something really BAD? I still can't get over the fact that it works fine and never occurs on the proverbial my machine... :) Thanks... Hoby -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Very strange problem with server and client software...
Hello. I am experiencing some very strange symptoms with some programs that communicate using ICS. Summary: The programs communicate fine (stable and consistent) when they are run on DIFFERENT machines. However, when run on the SAME machine, the communication between the programs is very unpredictable (sporadic failures, varying in frequency). In contrast to this, there are CERTAIN machines, where they DO run fine (stable and consistent) on the same machine. Oddly the working machines have Delphi installed on them and the failing machines do NOT have Delphi installed on them (which shouldn't matter). Detail: (sorry so long) I have implemented a thin client architecture using a basic Server / Client pattern. Meaning, there are two programs; one functions as a Server (actually a Broker) using TWSocketServer and another Client program uses TWSocket. The Client program initiates the connection and generates TCP traffic of a varying nature, depending upon specific user activities. The sessions are persistent (the client stays connected for the life of the application). The session traffic uses my own packetized protocol, something I have done many times (including with ICS). The Server uses a single TWSocketServer on the Main Form for all communications, meaning no Server based threads for TCP handling. However, while processing inbound packets from the Client program, it does hand some chores off to a thread pool that does the mundane chores (DB lookups, etc). A thread may or may not send one or more packets to the client, depending upon the particular protocol chore they were handling (requesting data of different types, reporting on Client interface activities, etc). The Server's outbound packet calls via the TWSocketClients all funnel through one critical section, which should provide thread safety for outbound traffic. There are a few other items that may worth noting. There is an additional load balancing program (Router) that is not directly engaged in the session. The Client first connects to the Router, which provides Server connection information to the Client. The Client disconnects from the Router and re-connects to the directed Server. There is an Admin channel between the Router and Server (Broker) that provides intelligent load balancing info, which OBVIOUSLY uses a SEPARATE socket interface, distinct from Client / Server channel. Additionally, this all occurs on top of my own Secure Channel implementation, which uses bi-directional BlowFish encryptors and multiple private keys to seed various encryption and randomizing algorithms. Also, I am using a MySQL Database via the MicroOLAP MySQLDAC components, which MicroOLAP claims is thread safe and appears generally to be. Oddly, the Client and Server programs work fine when both are running on my 2 development machines. (As a result, I didn't catch this issue until later in development, like before a demo. :-( ). However, when I try to run both of them on any other machine (so far), they communicate very poorly. Basically, they experience symptoms of random packet drops, but I don't think that is actually the case. I can't really tell, because I can not reproduce the scenario on a development machine where I could trace and debug the issue. The session will just hang, usually during the handshaking stage (Channel setup), but not always. However, if the Server and Client are on DIFFERENT machines, they work wonderfully, no problems at all, no matter what MACHINES they are on. I have tried a BUNCH of things to attempt to isolate and / or alleviate the problem. I have changed the Threading property on the Server Socket, which seems to have no effect. I have also tried using local MySQL Server vs. Remote MySQL server, no effect either way. As well as a many other tests / solutions. All to no avail. I am stumped in a big way. I understand that this is a really complicated implementation and my issue could be caused by a NUMBER of things. However, most of my testing seems to isolate the issue to the communications layer, for some unknown reason. I have never seen a scenario where the programs run fine from different machines, but won't work on the same machine. I am really hoping someone might have seen something similar? Could it be a WinSock bug? Any help would be appreciated. Thanks much. Hoby -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Very strange problem with server and client software...
for some unknown reason. I have never seen a scenario where the programs run fine from different machines, but won't work on the same machine. I am really hoping someone might have seen something similar? Could it be a WinSock bug? I don't think it is a winsock bug. The main difference between running programs on different machine and on the same machine is the speed. On the local machine, it is likely to be very fast. You have to pay attention to your DataAvailable event handler so that it doesn't care how data is split into packets. I suggest you use SocketSPY (see usermade page at ICS website) to spy on the communication between your programs, either when running on different computers or running on the same computer. You may modify SocketSpy slightly to log all traffic to a file foe later analysis and comparing running the program locally or remotely. -- Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html -- [EMAIL PROTECTED] http://www.overbyte.be - Original Message - From: Hoby Smith [EMAIL PROTECTED] To: twsocket@elists.org Sent: Friday, July 28, 2006 5:46 PM Subject: [twsocket] Very strange problem with server and client software... Hello. I am experiencing some very strange symptoms with some programs that communicate using ICS. Summary: The programs communicate fine (stable and consistent) when they are run on DIFFERENT machines. However, when run on the SAME machine, the communication between the programs is very unpredictable (sporadic failures, varying in frequency). In contrast to this, there are CERTAIN machines, where they DO run fine (stable and consistent) on the same machine. Oddly the working machines have Delphi installed on them and the failing machines do NOT have Delphi installed on them (which shouldn't matter). Detail: (sorry so long) I have implemented a thin client architecture using a basic Server / Client pattern. Meaning, there are two programs; one functions as a Server (actually a Broker) using TWSocketServer and another Client program uses TWSocket. The Client program initiates the connection and generates TCP traffic of a varying nature, depending upon specific user activities. The sessions are persistent (the client stays connected for the life of the application). The session traffic uses my own packetized protocol, something I have done many times (including with ICS). The Server uses a single TWSocketServer on the Main Form for all communications, meaning no Server based threads for TCP handling. However, while processing inbound packets from the Client program, it does hand some chores off to a thread pool that does the mundane chores (DB lookups, etc). A thread may or may not send one or more packets to the client, depending upon the particular protocol chore they were handling (requesting data of different types, reporting on Client interface activities, etc). The Server's outbound packet calls via the TWSocketClients all funnel through one critical section, which should provide thread safety for outbound traffic. There are a few other items that may worth noting. There is an additional load balancing program (Router) that is not directly engaged in the session. The Client first connects to the Router, which provides Server connection information to the Client. The Client disconnects from the Router and re-connects to the directed Server. There is an Admin channel between the Router and Server (Broker) that provides intelligent load balancing info, which OBVIOUSLY uses a SEPARATE socket interface, distinct from Client / Server channel. Additionally, this all occurs on top of my own Secure Channel implementation, which uses bi-directional BlowFish encryptors and multiple private keys to seed various encryption and randomizing algorithms. Also, I am using a MySQL Database via the MicroOLAP MySQLDAC components, which MicroOLAP claims is thread safe and appears generally to be. Oddly, the Client and Server programs work fine when both are running on my 2 development machines. (As a result, I didn't catch this issue until later in development, like before a demo. :-( ). However, when I try to run both of them on any other machine (so far), they communicate very poorly. Basically, they experience symptoms of random packet drops, but I don't think that is actually the case. I can't really tell, because I can not reproduce the scenario on a development machine where I could trace and debug the issue. The session will just hang, usually during the handshaking stage (Channel setup), but not always. However, if the Server and Client are on DIFFERENT machines, they work wonderfully, no problems at all, no matter what MACHINES they are on. I have tried a BUNCH of things to attempt to isolate and / or alleviate the problem. I have changed the Threading property on the Server Socket, which
Re: [twsocket] Very strange problem with server and client software...
Thanks for the info guys... :) I assumed the issue must somehow be related to sequencing issues impacted by the faster, local speeds, although exactly how I wasn't sure. On further testing, I have discovered something of interest. After setting the TWSocketServer MultiThreaded property to true, it appears to possibly fix the instability problem, but seems to inject another problem. With MultiThreaded set to true, I generated some additional log info from the Server app. Oddly, I found that the packet handling sequence order is altered in a particular handshaking process, causing the channel security syncing to fail. Here is some pseudo logic for what happens when MultiThreaded is set to True. Again, for some reason, this DOESN'T happen on my development machine, just on some other machines, and only when Server and Client are on the same machine: - Discrete portion of session logic flow during channel sync: 1. [SERVER] Send Packet to Client with random Challenge parameters. 2. [SERVER] Resync Server side of Client channel using new security parameters. 3. [CLIENT] Upon receipt of Challenge parameters from Server, resync Client side of channel using new parameters. 4. [CLIENT] Send Challenge response to Server using new channel parameters. 5. [SERVER] Receive and validate Client Challenge response, asserting trusted or untrusted channel state on pass / fail. The problem is that #2 MUST occur before #5. However, with MultiThreaded set to true, it does not occur in the correct order on some machines when both programs are running on the same machine. I don't quite understand why that would be, as #1 and #2 are basically consecutive lines of code, where #5 is a process that will be handled by a different thread function after receiving the client response packet. It exhibits symptoms like ProcessMessages is being called after #1, which receives and process the Client response before #2 can occur, although I am not calling ProcessMessages. Pseudo logic for the code fragment for #1 and #2 would be: - Thread function 1 Line 1. Generate Random challenge parameters. Line 2. Send Client Packet with info. Line 3. Resync Client Channel parameters. ... - Thread Function 2 (upon receipt of Client Challenge Response) If ValidateClientParms then Session = trusted else AbortSession. Question: Why would setting MultiThreaded to True cause Thread Function 2 to fire after Thread Function Line 2 which handles the response, before Thread Function Line 3 can execute? Why would this only occur with MultiThreaded set to true and only on some machines and not others? I can work around this by preparing the Server Challenge packet first, re-initializing the channel, then bypassing the channel overhead and sending the prepared packet raw. However, that would be a real kludge and I am more concerned that similar sequencing issues might surface in other areas. Thanks much for your time... Hoby Previous Francois response: I don't think it is a winsock bug. The main difference between running programs on different machine and on the same machine is the speed. On the local machine, it is likely to be very fast. You have to pay attention to your DataAvailable event handler so that it doesn't care how data is split into packets. I suggest you use SocketSPY (see usermade page at ICS website) to spy on the communication between your programs, either when running on different computers or running on the same computer. You may modify SocketSpy slightly to log all traffic to a file foe later analysis and comparing running the program locally or remotely. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Very strange problem with server and client software [solution]...
Hello. Well, ultimately, here is my solution. This appears to work 99.9% of the time, maybe 100%. There was one failure, but not sure if it was a related problem... ;) Oddly, turning on the TWSocket MultiThreaded property tends to alleviate the majority of symptoms. As I understand, the only major difference when MultiThreaded is on is that the ClientSocket utilizes its own messaging interface? Right? As for the issue I mentioned in the last email I posted, I solved the sequencing issue this way. Previously, I had a global send packet routine, wrapped in a Critical Section for thread safety. I use a single Global send routine so that I can gather statistics in one place (total bytes sent, total packets sent, total session bytes, etc). Since the MultiThreaded message loop was picking up responses before I was prepared to receive them, I created an additional couple of global routines. One is a global post packet routine, which processes the packet, but uses PutDataInSendBuffer to prepare and que up the outbound traffic (framing, encryption, etc), instead of sending it immediately. Then, when I am prepared to handle any related responses, I call a new global Send Buffer procedure, that simply issues a 0 byte Send on the ClientSocket to begin sending the que'd data. I then changed the sequence order sensitive handshaking routines to use the PostPacket / SendBuffer routines instead. This approach has appeared to fix the issue sufficiently. I am perplexed as to the nature of the issue itself, but for now, this will suffice. The scenario of both programs (client and server) being on one system is never going to occur during a live production environment. Solving this issue is really only relevant for development, testing and demonstration purposes. None-the-less, I am concerned that the surfacing of this issue might reflect some deeper logic flaw in the system as a whole. Does this solution sound appropriate? It appears to work fine. And does the problem issue sound like it should occur at all? Or does it sound like I have done something really BAD? I still can't get over the fact that it works fine and never occurs on the proverbial my machine... :) Thanks... Hoby -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
