Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
To close this one (sort of), The problem disappears when I use the OverbyteIcsHttpSrv.pas from the zipped daily snapshot (http://wiki.overbyte.be/arch/icsv8w.zip). regards, Merijn On 03/09/2014 23:34, Merijn Bosma wrote: Hi all, For a current project I need to build an HTTP server to which data can be POSTed, so I build one using OverbyteIcsWebServ.dpr as an example. I ran into stability problems, and starting slimming my code down to isolate the problem. I'm now at a point where it seems to be an issue in ICS code, though I haven't found the cause yet. Currently the issue is reproducible using the OverbyteIcsWebServ.dpr which comes with the ICS code as a demo. The issue is as follows: when posting a file to the OverbyteIcsWebServ, sometimes the post fails, on the client side this manifests as getting back a 408, on the server side it manifests as not receiving data at all, or missing the last block (in case the data is split up in blocks). Looking at OverbyteIcsWebServ.dpr, this means HttpServer1PostDocument() is called always and HttpServer1PostedData() is not called at all, or not for the last expected block. If you add the following line: Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); In procedure TWebServForm.HttpServer1PostedData() in OverbyteIcsWebServ1.pas, right after Inc(ClientCnx.FDataLen, Len); So it looks like this: Len := ClientCnx.Receive(ClientCnx.FPostedRawData + ClientCnx.FDataLen, Remains); { Sometimes, winsock doesn't wants to givve any data... } if Len <= 0 then Exit; { Add received length to our count } Inc(ClientCnx.FDataLen, Len); Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); { Check maximum length } if ClientCnx.FDataLen > MAX_UPLOAD_SIZE then begin When the fault situation happens, you will see log like: received 255 bytes, total received 255, total expecting 1420 received 255 bytes, total received 512, total expecting 1420 received 255 bytes, total received 768, total expecting 1420 received 255 bytes, total received 1024, total expecting 1420 received 255 bytes, total received 1280, total expecting 1420 Then it stops (as if the last block was never received), and timeout (error 408) occurs. Looking with wireshark shows that the last packet was on the wire, just not received in the application. To reproduce I've attached a small app which can be tested against OverbyteIcsWebServ.dpr.This application will show all files found in subdir 'postdata', you can double click one to post it to given URL. Sometimes it takes a long time to reproduce, sometimes it happens fast (first file you post). The 'machine gun' button in the app will post random files from the postdata subdir fast, to help reproduce. So, exact steps to reproduce are: - compile OverbyteIcsV8Gold.zip\Samples\Delphi\WebDemos\OverbyteIcsWebServ.dpr and run it - unzip HTTPPoster with the postdata subdir and run it - change the IP in HTTPPoster if required - hit the 'machine gun' button Log in HTTPPoster will show when the timeout occured (and the 'machine gun' will be shut off automatically). I'll try to dig into ICS code later, but first looks give me the idea that won't be really easy to debug, so I'm hoping for some help from the guru's thanks in advance, Merijn -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
> I've tried a number of times with this url > http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm?FileName=me > rijn.test > Can you please confirm if the upload went ok? > Received Post Data File, Size 4,550, Content Type: application/binary > FileName: merijn.test/cgi-bin/FileUpload/books.xml No, my application decodes the received POST data with TFormDataAnalyser, and that has the longer CGI file name. It ignores the URL argument if the HTTP content says multipart/form-data. It's a couple of years since I worked with this stuff and I don't POST files in any of my active applications, so I'm a little hazy on why it's there - possibly to test the form decoding components when someone in this mailing list was having a problem. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
Hi Angus, While I agree with you, the issue seems to be a little more complex. I've tried this connection over the internet, client on a remote location, server running locally. It still shows the same issue. What's ever more odd is that I switched the writing to TMemo.Lines with a sleep(), to see how long waiting is required to get things running. When running locally (both client and server on the same macine), a sleep(1) is enough to 'work around' the issue. When running the client on a remote site, I have to up this to sleep(100) to make things work (sleep(50) didn't work ok, didn't try between 50 and 100). So long story short, the slower the network connection, the longer the sleep has to be to get things working, which gives me the feeling it's not a normal 'two things happening at nearly the same time' kind of problem. Merijn On 04/09/2014 15:04, Angus Robertson - Magenta Systems Ltd wrote: No, I can't reproduce using that URL, I'm not really sure what triggers the situation I have seen cases where a bug that shows up during testing on a fast LAN is never reproduced on real world servers and networks. Try adding bandwidth limiting to your client to slow down the speed on the LAN. Set BandwidthLimit to bps, and add HttpoBandwidthControl to Options. This is an alternate to adding extra delays which is effectively what you are doing by updating the screen. There is probably a race condition somewhere, that has yet to be found. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
Hi Angus, I wasn't aware the /cg was in the filename, it was there unintentionally. I've tried a number of times with this url http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm?FileName=merijn.test Can you please confirm if the upload went ok? thanks, Merijn On 04/09/2014 16:30, Angus Robertson - Magenta Systems Ltd wrote: No, I can't reproduce using that URL You did not succeed in saving any files on the server, because my application ignores all upload file names with file path delimiters. It will be a better test if you strip off the path and just leave the XML file name. 12:42:59 Request: 192.168.1.162:80[id=5398] from 80.100.37.247 (bosma.xs4all.nl): http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm?FileName=merijn.test/cg i-bin/FileUpload/books.xml POST 12:42:59 Received Post Data File, Size 4,550, Content Type: application/binary 12:42:59 Illegal Upload File Name: merijn.test/cgi-bin/FileUpload/books.xml FileName: merijn.test/cgi-bin/FileUpload/books.xml FileTitle: Post URL: http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm >From IP Address: 80.100.37.247 12:42:59 Request Completed: 192.168.1.162:80[id=5398] 80.100.37.247 (bosma.xs4all.nl), Result 200, took 0ms, CurRead 4.97 Kbytes, CurWrite 2.35 Kbytes Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
> No, I can't reproduce using that URL You did not succeed in saving any files on the server, because my application ignores all upload file names with file path delimiters. It will be a better test if you strip off the path and just leave the XML file name. 12:42:59 Request: 192.168.1.162:80[id=5398] from 80.100.37.247 (bosma.xs4all.nl): http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm?FileName=merijn.test/cg i-bin/FileUpload/books.xml POST 12:42:59 Received Post Data File, Size 4,550, Content Type: application/binary 12:42:59 Illegal Upload File Name: merijn.test/cgi-bin/FileUpload/books.xml FileName: merijn.test/cgi-bin/FileUpload/books.xml FileTitle: Post URL: http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm >From IP Address: 80.100.37.247 12:42:59 Request Completed: 192.168.1.162:80[id=5398] 80.100.37.247 (bosma.xs4all.nl), Result 200, took 0ms, CurRead 4.97 Kbytes, CurWrite 2.35 Kbytes Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
> No, I can't reproduce using that URL, I'm not really sure what > triggers the situation I have seen cases where a bug that shows up during testing on a fast LAN is never reproduced on real world servers and networks. Try adding bandwidth limiting to your client to slow down the speed on the LAN. Set BandwidthLimit to bps, and add HttpoBandwidthControl to Options. This is an alternate to adding extra delays which is effectively what you are doing by updating the screen. There is probably a race condition somewhere, that has yet to be found. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
Duh, looked at the page but didn't real bottom lines :\ sorry for that. No, I can't reproduce using that URL, I'm not really sure what triggers the situation (see the fact that calling TMemo.Lines.Add() seems to avoid the problem). I'm currently adding debug log into ICS, and it seems that at the moment things go wrong, TCustomWSocket.Do_FD_WRITE is called when it should not. Also in fault situation wParam is 668 (where normally it's 652). Don't know yet why or what it means though. Cause Do_FD_WRITE is called, some flags are switched in OverbyteIcsHttpSrv, which makes that the last block is dropped. Merijn On 04/09/2014 13:39, Angus Robertson - Magenta Systems Ltd wrote: Thanks for the idea, can I upload files from software instead of using the webpage? This is explained when you read the web page. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
> Thanks for the idea, can I upload files from software instead of > using the webpage? This is explained when you read the web page. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
Hi Angus, Thanks for the idea, can I upload files from software instead of using the webpage? Thanks, Merijn On 04/09/2014 13:25, Angus Robertson - Magenta Systems Ltd wrote: For a current project I need to build an HTTP server to which data can be POSTed, so I build one using OverbyteIcsWebServ.dpr as an example. I ran into stability problems, and starting slimming my code down to isolate the problem. Please try uploading your file to one of my public servers, which uses the ICS webapp server component, and see if you get the same problem. http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
> For a current project I need to build an HTTP server to which data > can be POSTed, so I build one using OverbyteIcsWebServ.dpr as an > example. > I ran into stability problems, and starting slimming my code down > to isolate the problem. Please try uploading your file to one of my public servers, which uses the ICS webapp server component, and see if you get the same problem. http://www1.telecom-tariffs.co.uk/testing/uploadfile.htm Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
I have some new info, which unfortunately only makes things less logical for me :p Since at the moment the focus, for me, is moved from my code to ICS code, I've switched back to a simple HTTP server I made in stead of the demo. This server app is much smaller then the demo, thus easier to debug, and shows exactly identical results to the demo app. The complete source of my small server app is at the bottom of this post. Here is some real logging from both client and server side, when the problem occurs: client: 08:30:56.149 - starting machine gun 08:30:56.164 - done. errcode: 0 (No Error) statuscode: 200 08:30:56.240 - done. errcode: 0 (No Error) statuscode: 200 08:30:56.325 - done. errcode: 0 (No Error) statuscode: 200 08:31:06.424 - problem occured, stopping machine gun 08:31:06.432 - machine gun stopped 08:31:06.448 - done. errcode: 8 (Not a Winsocket error) statuscode: 404 server: 08:30:56.155 - post document application/binary 08:30:56.159 - rx 1059 / 1059 08:30:56.226 - post document application/binary 08:30:56.235 - rx 4550 / 4550 08:30:56.300 - post document application/binary 08:30:56.310 - rx 4550 / 4550 08:30:56.385 - post document application/binary 08:30:56.393 - rx 1630 / 4550 08:30:56.400 - rx 2920 / 4550 The interesting part is this: if I uncomment the Log() call in TForm1.OnClientConnect, the problem does not occur anymore. It doesn't matter what is logged, as long as something is added to Form1.MemoLog.Lines, the problem does not occur. Looking into TMemo, nothing much seems to happen besides some messages being sent: procedure TMemoStrings.Insert(Index: Integer; const S: string); var SelStart, LineLen: Integer; Line: string; begin if Index >= 0 then begin SelStart := SendMessage(Memo.Handle, EM_LINEINDEX, Index, 0); if SelStart >= 0 then Line := S + #13#10 else begin SelStart := SendMessage(Memo.Handle, EM_LINEINDEX, Index - 1, 0); if SelStart < 0 then Exit; LineLen := SendMessage(Memo.Handle, EM_LINELENGTH, SelStart, 0); if LineLen = 0 then Exit; Inc(SelStart, LineLen); Line := #13#10 + s; end; SendMessage(Memo.Handle, EM_SETSEL, SelStart, SelStart); SendMessage(Memo.Handle, EM_REPLACESEL, 0, Longint(PChar(Line))); end; end; Sending these messages in TForm1.OnClientConnect instead of adding something to the memo makes the problem come back again. Hoping for some insight. regards, Merijn unit main; interface uses Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls, OverbyteIcsHttpSrv, OverByteIcsWSocketS; type TMyHTTPConnection = class(THttpConnection) private fData: string; fAllReceived: boolean; procedure InternalOnPostedData(Sender: TObject; ErrCode: word); public procedure Init; end; TMyHTTPServer = class(THttpServer) protected procedure WSocketServerClientConnect(Sender: TObject; Client: TWSocketClient; Error: Word); override; procedure TriggerPostDocument(Sender: TObject; var Flags: THttpGetFlag); override; end; TForm1 = class(TForm) MemoLog: TMemo; procedure FormCreate(Sender: TObject); procedure FormDestroy(Sender: TObject); private fMyHTTPServer: TMyHTTPServer; procedure OnClientConnect(Sender: TObject; Client: TObject; Error: word); public procedure Log(st: string); end; var Form1: TForm1; implementation {$R *.dfm} { TMyHTTPConnection } procedure TMyHTTPConnection.Init; begin fData := ''; fAllReceived := false; Self.OnPostedData := InternalOnPostedData; end; procedure TMyHTTPConnection.InternalOnPostedData(Sender: TObject; ErrCode: word); var Client: TMyHTTPConnection; Data: string; Flags: THttpGetFlag; begin Client := TMyHTTPConnection(Sender); Data := Client.ReceiveStr(); if fAllReceived then begin sleep(0); exit; end; fData := fData + Data; Form1.Log(Format('rx %d / %d', [Length(fData), Client.RequestContentLength])); if (Length(fData) = Client.RequestContentLength) then begin fAllReceived := true; PostedDataReceived(); AnswerString(Flags, '', '', '', 'empty'); end; end; { TMyHTTPServer } procedure TMyHTTPServer.TriggerPostDocument(Sender: TObject; var Flags: THttpGetFlag); begin inherited; form1.Log('post document ' + TMyHTTPConnection(Sender).RequestContentType); TMyHTTPConnection(Sender).LineMode := false; TMyHTTPConnection(Sender).fData := ''; Flags := hgAcceptData; end; procedure TMyHTTPServer.WSocketServerClientConnect(Sender: TObject; Client: TWSocketClient; Error: Word); begin inherited; TMyHTTPConnection(Client).Init(); end; procedure TForm1.FormCreate(Sender: TObject); begin fMyHTTPServer := TMyHTTPServer.Create(nil); fMyHTTPServer.OnClientConnect := OnClientConnect; fMyHTTPServer.ClientClass := TMyHTTPConnection; fMyHTTPServer.DocDir := ''; fMyHTTPServer.Port := '80'; fMyHTTPServer.Addr := '0.0.0.0'; fMyHTTPServer.Start(); end; procedure TForm1.FormDestroy(Sender: TObject)
Re: [twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
I just noticed I can't attach a file through the mailing list (makes sense), so I've put the zip file with HTTPPoster sources online: http://www.xs4all.nl/~bosma/HTTPPoster.zip regards, Merijn On 03/09/2014 23:34, Merijn Bosma wrote: Hi all, For a current project I need to build an HTTP server to which data can be POSTed, so I build one using OverbyteIcsWebServ.dpr as an example. I ran into stability problems, and starting slimming my code down to isolate the problem. I'm now at a point where it seems to be an issue in ICS code, though I haven't found the cause yet. Currently the issue is reproducible using the OverbyteIcsWebServ.dpr which comes with the ICS code as a demo. The issue is as follows: when posting a file to the OverbyteIcsWebServ, sometimes the post fails, on the client side this manifests as getting back a 408, on the server side it manifests as not receiving data at all, or missing the last block (in case the data is split up in blocks). Looking at OverbyteIcsWebServ.dpr, this means HttpServer1PostDocument() is called always and HttpServer1PostedData() is not called at all, or not for the last expected block. If you add the following line: Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); In procedure TWebServForm.HttpServer1PostedData() in OverbyteIcsWebServ1.pas, right after Inc(ClientCnx.FDataLen, Len); So it looks like this: Len := ClientCnx.Receive(ClientCnx.FPostedRawData + ClientCnx.FDataLen, Remains); { Sometimes, winsock doesn't wants to givve any data... } if Len <= 0 then Exit; { Add received length to our count } Inc(ClientCnx.FDataLen, Len); Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); { Check maximum length } if ClientCnx.FDataLen > MAX_UPLOAD_SIZE then begin When the fault situation happens, you will see log like: received 255 bytes, total received 255, total expecting 1420 received 255 bytes, total received 512, total expecting 1420 received 255 bytes, total received 768, total expecting 1420 received 255 bytes, total received 1024, total expecting 1420 received 255 bytes, total received 1280, total expecting 1420 Then it stops (as if the last block was never received), and timeout (error 408) occurs. Looking with wireshark shows that the last packet was on the wire, just not received in the application. To reproduce I've attached a small app which can be tested against OverbyteIcsWebServ.dpr.This application will show all files found in subdir 'postdata', you can double click one to post it to given URL. Sometimes it takes a long time to reproduce, sometimes it happens fast (first file you post). The 'machine gun' button in the app will post random files from the postdata subdir fast, to help reproduce. So, exact steps to reproduce are: - compile OverbyteIcsV8Gold.zip\Samples\Delphi\WebDemos\OverbyteIcsWebServ.dpr and run it - unzip HTTPPoster with the postdata subdir and run it - change the IP in HTTPPoster if required - hit the 'machine gun' button Log in HTTPPoster will show when the timeout occured (and the 'machine gun' will be shut off automatically). I'll try to dig into ICS code later, but first looks give me the idea that won't be really easy to debug, so I'm hoping for some help from the guru's thanks in advance, Merijn -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] issues with POSTing data to ICS HTTP server (reproducible with ICS HTTP server demo)
Hi all, For a current project I need to build an HTTP server to which data can be POSTed, so I build one using OverbyteIcsWebServ.dpr as an example. I ran into stability problems, and starting slimming my code down to isolate the problem. I'm now at a point where it seems to be an issue in ICS code, though I haven't found the cause yet. Currently the issue is reproducible using the OverbyteIcsWebServ.dpr which comes with the ICS code as a demo. The issue is as follows: when posting a file to the OverbyteIcsWebServ, sometimes the post fails, on the client side this manifests as getting back a 408, on the server side it manifests as not receiving data at all, or missing the last block (in case the data is split up in blocks). Looking at OverbyteIcsWebServ.dpr, this means HttpServer1PostDocument() is called always and HttpServer1PostedData() is not called at all, or not for the last expected block. If you add the following line: Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); In procedure TWebServForm.HttpServer1PostedData() in OverbyteIcsWebServ1.pas, right after Inc(ClientCnx.FDataLen, Len); So it looks like this: Len := ClientCnx.Receive(ClientCnx.FPostedRawData + ClientCnx.FDataLen, Remains); { Sometimes, winsock doesn't wants to givve any data... } if Len <= 0 then Exit; { Add received length to our count } Inc(ClientCnx.FDataLen, Len); Display(Format('received %d bytes, total received %d, total expecting %d', [Len, ClientCnx.FDataLen, ClientCnx.RequestContentLength])); { Check maximum length } if ClientCnx.FDataLen > MAX_UPLOAD_SIZE then begin When the fault situation happens, you will see log like: received 255 bytes, total received 255, total expecting 1420 received 255 bytes, total received 512, total expecting 1420 received 255 bytes, total received 768, total expecting 1420 received 255 bytes, total received 1024, total expecting 1420 received 255 bytes, total received 1280, total expecting 1420 Then it stops (as if the last block was never received), and timeout (error 408) occurs. Looking with wireshark shows that the last packet was on the wire, just not received in the application. To reproduce I've attached a small app which can be tested against OverbyteIcsWebServ.dpr.This application will show all files found in subdir 'postdata', you can double click one to post it to given URL. Sometimes it takes a long time to reproduce, sometimes it happens fast (first file you post). The 'machine gun' button in the app will post random files from the postdata subdir fast, to help reproduce. So, exact steps to reproduce are: - compile OverbyteIcsV8Gold.zip\Samples\Delphi\WebDemos\OverbyteIcsWebServ.dpr and run it - unzip HTTPPoster with the postdata subdir and run it - change the IP in HTTPPoster if required - hit the 'machine gun' button Log in HTTPPoster will show when the timeout occured (and the 'machine gun' will be shut off automatically). I'll try to dig into ICS code later, but first looks give me the idea that won't be really easy to debug, so I'm hoping for some help from the guru's thanks in advance, Merijn -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be