[twsocket] THttpCli.GetAsync without any time-out implementation?
Hello! Two simple questions, both related: (1) Is it OK to create a new THttpCli component and call GetAsync from an THttpCli.OnDocData event handler? (2) If I do THttpCli.GetAsync *without* any kind of time-out mechanism, what's the worst thing that might happen? Will the component run for ever or will it stop when the TCP/IP stack gives up? Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Posting data to web server using THttpCliunder usercontrol
Francois PIETTE wrote: Using hgWillSendMySelf I noticed the proxy server immediately connects the client to the server and the connection is kept open as long as I'm sending data. If the proxy server does the same for the long HTTP post it would provide a very valuable tool for what I need. I wouldn't base my software on this behaviour which is defenitely not guaranteed. HTTP protocol is a stateless protocol and you have poor connection control. Just ask for the connection to be kept alive and hope it is. If it isn't, then it will be reopened anyway by the component. None of the things I'm doing expects the HTTP protocol to be anything but stateless. I'll re-explain how my downstream (that is - HTTP download) connection works. The client requests a document from my web server. It does this going through the proxy. The server response (200 OK), does NOT send an Content-length header and starts sending the document very very slowly, but it keeps sending for ever. It looks just like a large file download over a very slow link. Since there's no Content-length header the proxy is unable to determine if the document has finished downloading or not, it will keep receiving the given document for as long as the server sends. I noticed the proxy does one other nice thing: it immediately relays all received data to the client, and it really makes sense. Real-life example that should force the proxy to behave in this very same manner: The client is downloading a 4Gb ISO image through the proxy. The proxy can't possibly wait for the WHOLE download to finish before starting to relay data to the client, it just doesn't make sense. It must start sending data to the client as soon as data is available. What I'm thinking about is using the very same thing the other way around: Doing a slow POST! I do understand downloading and uploading are governed by different rules, so I'd like to try this too. If you still say it's unlikely to work, I'll stop trying (by now I'm sure I explained things clearly enough so there's no misunderstanding). On a different note, how do programs like zebedee provide a TCP/IP tunnel through a HTTP proxy? I understand how the downstream part works, but how about the upstream part? This is quite easy. You can send the command CONNECT to a HTTP proxy and then the proxy will work transparently as a simple relay, without ever trying to understand what you send in the stream. This is how HTTPS works thru the proxy. Since HTTPS encrypt everything, the proxy is unable to do anything. It can just relay the data. That's the purpose of the CONNECT command. I might try implementing this as well. Trying an CONNECT first and, if that doesn't work, fall-back to alternative, slower methods. Can the THttpCli component be used for proxy CONNECT only, bypassing everything else? Or should I start with a vanilla TWSocket and work my way from that? Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Posting data to web serverusing THttpCliunder usercontrol
Francois Piette wrote: This is proxy implementation dependent. The proxy could buffer the document up to some point and then give up and forward everything. It could also handle document differently according to the content-type. If you consider this would not happend, then it's OK for me. I'm just giving you the ideas I have :-) Thanks for the input, it's good to know how things might fail :) For my downstream connection I'll use the following logic (when proxy is enabled): (a) Try CONNECT; If it fails go to (b) (b) Start my standard HTTP Wizardry; At the first NOP received send an ACK to the server, letting it know I actually got the message. If the server has a real message to send (ie: not a NOP) and it hasn't received any NOP-confirmation yet, I'll send the message and then close the connection. If the server has already received an ACK for a NOP when it needs to send the real message (and the ACK was received in a timely manner) I'll know the proxy is relaying all sent data in a timely manner and I'll send the real message without closing the connection. In my opinion this should work, no matter how the server is configured. It should also work if I'm dealing with an transparent proxy (so I don't actually know I'm dealing with a proxy). Do you see any wholes in my logic? Unfortunately for my upstream connection I only have two options: use CONNECT or use successive POST's; The successive POSTs variant doesn't sound all that good, since I also want to tunnel interactive traffic through the HTTP proxy (interactive=VNC, not plain-old chat; Chat would pose no problem) Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] HttpCli ContentRangeBegin
You know, modern hard drives actually checksum written data so you never get corrupted data back. You might get NO data back (that is, an I/O error) but that would make any operation on the file fail, including burning the file to a CD. I've read this in a howto for Linux'es software raid driver. That's where they also say the BUS that data travels to the HDD is NOT checksummed and that has a grater probability of introducing an error! Also there are two kinds of checksums applied to those large downloads. There's the MD5 checksum that can be reproduced by the man in the middle so it's useless for security (after all, if the man in the middle is willing to waist bandwidth to fake the main download, what does it take to also fake the checksum?). And then there are the ASC checksums, I think those are used for security as well. Jack wrote: It is actually for data integrity as well (more than security, in my opinion.) When it comes to large file download, there might be corrupted bytes. Then this is more likely caused by HD errors then network errors. Conclusion: I think data corruption might be a problem in some cases. Notice how all Linux distributions include MD5 hashes for all downloads, so they can be checked on the receiving end? This is not to detect data corrumption because of data transmission but to detect man in the middle attack. MD5 checksum allow the user to check if the data file he downloaded is the same as the data file the developper dropped on the server and was not replaced either on the server or by someone intercepting the communication. In think in the context you mention, MD5 is used for security, no for data integrity. -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Posting data to web server using THttpCli under user control
Hello everyone. I'm using THttpServer to send a document myself (using hgWillSendMySelf) so I can keep sending new data without requiring a new connection. It works just fine. Now I want to do the same the other way around: I want to use THttpCli to continually post data, without closing the connection. Is it possible? Will it work through an HTTP proxy? Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] What might cause ICS to fail and we need to be aware of?
Francois Piette wrote: I don't know of ANY component where it is safe to call the message pump from one of its event without knowing what happend. Even for a simple TButton.OnClick, you can get strange result if you call the message pump within the OnClick handler because the handler is re-entered. The problem is more visible with TWSocket since it is likely the event is always reentered because of the high rate network I/O has. That's not entirely true. Delphi's GUI model is event-driven so I'm willing to bet ANY call to Application.ProcessMessages will be made as a result of handling an message or event. And I'm willing to bet most calls to Application.ProcessMessages can be traced to ether an TButton.OnClick event or a menu item's OnClick! And if it's such a big issue for ICS, why isn't there a simple 5-line-of-code test that would raise an exception if any code is re-entered? Besides, I'm asking about such behavior because of the way the THttpCli component behaves for me. If you remember one of my earlier questions, I had lots of problems with the component failing to connect to my HTTP server. Using Ethereal I determined the connection is poor (lost packets, duplicate ACK's) BUT the component still averages an too high number of failed connections. I don't think I've had 5 consecutive sessions where no connections timed-out. And I'm saying the failure rate is too high because I never saw such a problem using my web browser. Not once! And after the time out expires and the component aborts and restarts it's Get, it usually finishes very very quickly. So I need to ask: Is there some other obvious thing I'm missing, like a call to Application.ProcessMessage? What else should I be looking for. -- Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] What might cause ICS to fail and we need to be aware of?
Francois Piette wrote: Because it would need to create a try/finally frame which would have an impact on the performance given the high rate the events may be triggered when speaking about network I/O. And it is very easy for the component user to use a flag to detect reentrancy in his own code. A try/finally frame does not have a significant impact on performance IF no exception is actually raised. I've done some tests on my 2,21Ghz AMD 64 X2: adding a standard frame adds about 0.2312 milliseconds to a call. In other words, 1 millisecond every 5000 calls. I've done the measuring in a loop that ran for 1 (that's 100 million times so you don't need to count the zero's). Anyway, the vast majority of message pump call within an event handler is a result of the developper not understanding how windows is working. Those developers add ProcessMessages call to have the screen refreshed or similar issues. That is frequently not the correct way to do it and it give strange results in many cases (for example when a user double-click on a simple button instead of simple click). That's true. Besides, I'm asking about such behavior because of the way the THttpCli component behaves for me. If you remember one of my earlier questions, I had lots of problems with the component failing to connect to my HTTP server. Using Ethereal I determined the connection is poor (lost packets, duplicate ACK's) BUT the component still averages an too high number of failed connections. I don't think I've had 5 consecutive sessions where no connections timed-out. And I'm saying the failure rate is too high because I never saw such a problem using my web browser. Not once! And after the time out expires and the component aborts and restarts it's Get, it usually finishes very very quickly. So I need to ask: Is there some other obvious thing I'm missing, like a call to Application.ProcessMessage? What else should I be looking for. Search for all ProcessMessages and ask yourself if it is really needed. Then remove it. Thinks about all indirect calls to ProcessMessages such as displaying any modal dialog box or form. All ProcessMessages calls are accounted for; None would run from within an ICS event handler. And if you must know what I'm using ProcessMessages for I'll tell you: I'm running my own ShowModal variant because the built-in one is not flexible enough. -- Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Posting data to web server using THttpCli under usercontrol
Francois Piette wrote: I'm using THttpServer to send a document myself (using hgWillSendMySelf) so I can keep sending new data without requiring a new connection. It works just fine. Now I want to do the same the other way around: I want to use THttpCli to continually post data, without closing the connection. You can have a HTTP post which never ends. But probably it is not what you really want. Since you control the client part, why not send as many HTTP requests as you need ? Using hgWillSendMySelf I noticed the proxy server immediately connects the client to the server and the connection is kept open as long as I'm sending data. If the proxy server does the same for the long HTTP post it would provide a very valuable tool for what I need. If it doesn't work I'll simply use as many HTTP posts as necessary. And if it makes any difference, for this part of my application I control both the client and the server; It's the potential proxy in the middle I'm not controlling. I need to use HTTP to make sure it works through the proxy. On a different note, how do programs like zebedee provide a TCP/IP tunnel through a HTTP proxy? I understand how the downstream part works, but how about the upstream part? Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Multi-Threaded THttpServer?
Hello! Here's me again, trying to do strange things. I'm working on a HelpDesk application that should include a Chat function, amongst other things. I want to do it all using HTTP only (that is, no direct connection, everything needs to be pure HTTP). I really want this HTTP-only thing because I want my application to work in places where my clients only have access to the Internet using an HTTP Proxy! The chat application has 2 basic components: send your text, receive text sent by the other party. The send your text is really easy, but the receive text sent by the other party part is a bit more difficult, because I can't keep an open connection between client and server, I need the client to pool for text sent from the server! That is, the client will GET a document of the following format: http://myserver.server.ro/getchat?conversation=12346seq=1 The server should return any available text for the given conversation or an NOP if no text is available. But here's a trick: If there's no text available for the connection I would like to delay returning an NOP until there IS some text available, or until a 10 seconds delay elapses. This would stop the client from going into a bandwidth-consuming busy-loop. Unfortunately THttpServer doesn't include a MultiThreaded checkbox like TSocketServer does, and I'm not sure what I should do to Sleep() without actually freezing the server in the process! I might try subclassing THttpServer and setting FWSocketServer.MultiThreaded = True in CreateSocket but I know too little about the internals of THttpServer and ICS in general to understand the consequences of doing this. Any help on the matter is welcomed, thanks. -- Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Multi-Threaded THttpServer?
Dan wrote: You could either: (1) Not reply to the client until after 10 seconds (using a TTimer?). You don't have to sleep. I'm sure the HTTP server has some kind of delayed reply mechanism. You can put the client in a 'waiting' list and only send the reply after your timeout. That would be plain simply perfect for what i need but how do I do that? For the moment I only know how to use the OnGetDocument to generate the response and it doesn't provide any room for delaying it's processing! At what point in time, or where from can I put a request on hold so it doesn't get processed normally? Please note I need to have at least part of the processing go normally, so I actually know it's a valid HTTP request and I know what document the user is requesting etc. (2) Have the client only retry after 10 seconds. There is a header you can send to get the client to refresh to a new URL after so many seconds. Many sites use it for redirects...they redirect after 5 seconds for example. That would be easy to implement but would not solve the problem (at least not in an acceptable way). In this way the client would receive messages in batches every 10 seconds (or whatever the retry time is). If I have the server do the waiting the messages will flow much faster. Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Cosmin Prund Sent: 18 November 2006 12:55 To: ICS support mailing Subject: [twsocket] Multi-Threaded THttpServer? Hello! Here's me again, trying to do strange things. I'm working on a HelpDesk application that should include a Chat function, amongst other things. I want to do it all using HTTP only (that is, no direct connection, everything needs to be pure HTTP). I really want this HTTP-only thing because I want my application to work in places where my clients only have access to the Internet using an HTTP Proxy! The chat application has 2 basic components: send your text, receive text sent by the other party. The send your text is really easy, but the receive text sent by the other party part is a bit more difficult, because I can't keep an open connection between client and server, I need the client to pool for text sent from the server! That is, the client will GET a document of the following format: http://myserver.server.ro/getchat?conversation=12346seq=1 The server should return any available text for the given conversation or an NOP if no text is available. But here's a trick: If there's no text available for the connection I would like to delay returning an NOP until there IS some text available, or until a 10 seconds delay elapses. This would stop the client from going into a bandwidth-consuming busy-loop. Unfortunately THttpServer doesn't include a MultiThreaded checkbox like TSocketServer does, and I'm not sure what I should do to Sleep() without actually freezing the server in the process! I might try subclassing THttpServer and setting FWSocketServer.MultiThreaded = True in CreateSocket but I know too little about the internals of THttpServer and ICS in general to understand the consequences of doing this. Any help on the matter is welcomed, thanks. -- Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Multi-Threaded THttpServer?
Francois PIETTE wrote: You defenitely doin't need multithread to throttle a connexion. See how it is done in the HTTP client component where you have a Bandwidth limitation feature. I'm not interested in throttling a connection. I don't have an set amount of data and want to send it a slower rate. I just want to delay answering a request for a while, hoping I'll have an better answer a few seconds later. This is a bit of pseudo code for what I think I'll be able to do in OnGetDocumnet if OnGetDocument would be a blocking thread. I'm not saying this is the only way to do it, but this is what I want to do: function GetNextMessage(Connection):string; var MaxDelay:TDateTime; begin MaxDelay := Now + EncodeTime(0, 0, 10, 0); while (not Connection.MessagesAvailable) and (Now MaxDelay) do Sleep(100); if Connection.MessagesAvailable then Result := Connection.ConcatenatedMessages else Result := 'NOP'; end; Well... since starting writing this message I received an other Francoise PIETTE answer to my question on the mailing list (the hgWillSendMySelf answer) and that provides a much better solution to my problem! I can now imagine better ways of exploiting the HTTP protocol in order to get my message across, without threads and blocking communications. Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] Multi-Threaded THttpServer?
Francois PIETTE wrote: (1) Not reply to the client until after 10 seconds (using a TTimer?). You don't have to sleep. I'm sure the HTTP server has some kind of delayed reply mechanism. Yes, you can reply later. Use hgSendMySelf Does it work like this? In my OnGetDocument I set Flags to hgWillSendMySelf and then use the client component's send method do send() the data, then use it's Shutdown() method to gracefully close the connection and let the other side know I've finished sending? Or is there something more to this, like taking into account Keep Alive connections? I can figure it out myself from here, thanks for the tip! Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] HttpCli ContentRangeBegin
Francois PIETTE wrote: (3) In the code running after a failed download I'm removing the last portion of the received data, just in case it's corrupted. I noticed this behavior in a freeware download manager I used to use some time ago. But now I'm asking: is this really necessary? HTTP traffic travels over TCP and TCP is supposed to be an checksummed, reliable protocol. Is the probability of receiving corrupt data high enough to make such radical surgery useful? If so, is tail-trimming the best solution, or should I implement some other kind of checksumming to make sure no portion of the code is actually corrupt (after all, corruption might occur anywhere in the document, not just in the last received portion). You are right, I don't see any reason to throw away part of the received data. By TCP protocol specification, only valid data is delivered to the application. If you get it, it is correct data. I did a bit of googling on TCP, IP and transport layer checksumming and found some interesting results. It seems that TCP provides an additive checksum to protect it's payload, and apparently that's not very strong. Transport layers provide allot stronger passwords: Ethernet uses 32 bit CRC checksums, PPP uses 16 bit CRC checksums and ATM uses a 8 bit CRC checksum that only protects the header. The weakest link from HTTP server to CLIENT is probably the link from CLIENT to ISP. The worst case scenario would be a dial-up connection using PPP-only. A better but still bad scenario would be an ADSL connection running PPP-over-ATM (because ATM does not seem to checksum it's payload). Most of my clients would fit in one of those 2 groups! Considering those facts, throwing away part of the already received data might be a good thing IF the connection to the HTTP server was lost because the client lost it's connection to the ISP. Chances are the connection to the ISP was lost because the line became noisy. If the connection to the HTTP server was lost on grounds of network congestion then throwing away part of the received data would be meaningless since the chances for data corruption are equal all over the received data, they're not higher towards the end of the file. Conclusion: I think data corruption might be a problem in some cases. Notice how all Linux distributions include MD5 hashes for all downloads, so they can be checked on the receiving end? I decided to implement MD5-based file checking for my downloads (my application only downloads stuff from my own site, so I've got everything under my control). I've done this because I know I've got quite a few clients on very bad dial-up lines. If I didn't have those clients I would have done no checking at all. -- Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] HttpCli ContentRangeBegin
Hello. I've build a download manager module for my application arround THttpCli and I'm making use of ContentRangeBegin to resume failed downloads. It all works very nice and smooth, and all my downloads resume fine. Unfortunately my downloads only fail when I unplug the network cable from my computer OR when I disable my network adapter... and I don't think those are the most common causes of disconnections for the average user! So I'll have to ask about abnormal situations, just to make sure I'm getting things right: (1) What happens if I use ContnetRangeBegin and the server doesn't support ranges (ex: if I'm downloading the results of a script)? Will I end up with the whole document append to the already downloaded portion of the document? (2) I noticed there's a AcceptRanges property that's set up by THttpCli from the headers it receives while processing a request. I supose this is server-dependent. Can I make use of this to determine if ranges are supported? (3) In the code running after a failed download I'm removing the last portion of the received data, just in case it's corrupted. I noticed this behavior in a freeware download manager I used to use some time ago. But now I'm asking: is this really necessary? HTTP traffic travels over TCP and TCP is supposed to be an checksummed, reliable protocol. Is the probability of receiving corrupt data high enough to make such radical surgery useful? If so, is tail-trimming the best solution, or should I implement some other kind of checksumming to make sure no portion of the code is actually corrupt (after all, corruption might occur anywhere in the document, not just in the last received portion). Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Normal processing and abnormal(errors) processing for THttpCli
Hello everyone. I'm having good progress using the THttpCli component, but now I need to have a few things cleared in my mind: (1) I noticed THttpCli component always triggers it's OnRequestDone event when it finishes processing, no matter what caused this processing to stop (successfully finished downloading the document, stopped because of network error, stopped because I called Abort). Is this true? Is there a situation where processing finished but OnRequestDone is NOT called? Or the reverse, a situation where OnRequestDone is called but the component is not done? (2) Can I use the ErrCode parameter of OnRequestDone as an reliable indicator of normal processing? I need to distinguish between normal request done and request done because of anything else. Abnormal request need to be retried, making use of HttpCli's ContentRangeBegin... (3) If ErrCode is not an good indicator of error/success, is HttpCli.StatusCode an good indicator? (my guess is this not good) (4) I noticed HttpCli.State = httpReady when the component is ready. Is State ever httpReady in the course of normal processing? My questions might be a bit dumb but I'm new to the concept of async processing in ICS and I need to make sure I get it right! Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
[twsocket] Help on ICS's HTTP client component
Hello everyone. Note: This is a re-post of a message sent to the Talk list, with more detail on the problem. This message gets sent the second time to this list too. I sent the first one before receiving elist's subscription confirmation so I don't think it went through. I'm having some trouble using the THttpCli component from the ICS suite. I'm using this component to simply download a document off a web site (the document might be a configuration file or a compressed DLL file or whatever). It mostly works but some times it just stops (gets stuck in a state - the last OnChangeState message I'm getting is httpDnsLookupDone) and then nothing happens. My application's GUI doesn't get stuck because Application's message pump is working, but the download doesn't finish ether. Or, if something does happen, I know nothing about it... Please note I'm using GetAsync to get my document. Extra information: I'm using the version of ICS marked [ Download the latest ICS-V5 Distribution ] on http://www.overbyte.be/frame_index.html. Is V6 for general consumption? Should I try that? The web server I'm accessing is likely not the cause for this problem as the server is an Apache server running on a Windows machine on my LAN. I did implement a kind of time-out mechanism into my download thing: if no event is triggered for a given amount (no state change, no data received, no error) I consider the connection to be timed out. This gets my application back on the track working properly but this feels like a hack, it doesn't seem to be the right thing to do. Please correct me if I'm wrong! Thanks, Cosmin Prund -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
