On Tue, 2008-02-12 at 15:24 -0500, Sam Berlin wrote: > Whew! Long subject line... > > I've been looking into exactly how all these components interact, and > it got me to wondering: how exactly are people using these objects (if > anyone is) right now?
Hi Sam I suspect none really uses HttpCore NIO for pure client side HTTP stuff. It is mostly being used to implement HTTP proxies > The example NHttpClient is very simplistic in > that it just sends a basic GET request for "/" to the three hosts and > accepts any response. Trying to turn the sample into a somewhat more > real-world scenario (say, a crawler) would seem to involve placing a > lot more information in the context of each request/response & > attachments. For instance, expanding the example into a crawler would > require: > > 1) On handleResponse, it parses the body for more links and adds them > as potential outgoing requests in the context. > 2) handleResponse somehow (?) IOControl#requestOutput will eventually cause the NHttpClientHandler#requestReady to fire, which in its turn will invoke HttpRequestExecutionHandler#submitRequest. > triggers another submitRequest to be > called with the right context. > 3) submitRequest looks up the new context information and submits > more requests. > 4) submitRequest could limit the number of pipelined attempts to a > given host by storing more data in the context and > incrementing/decrementing the concurrent attempts, which > handleResponse would need to manage. > > It becomes a little harder to make it work if you want to use a > variable number of connections and share the context. I imagine there > would need to be some kind of CrawlContext that's shared as the > attachment among multiple connects and used within > handleResponse/submitRequest. > > So... my question is, is this line of thought correct? Am I thinking > about the ConnectingReactor & ExecutionHandler the wrong way? I think this line of thought is correct. However, several major pieces of functionality are still missing, namely a connection manager and a pipelining capable NHttpClientHandler, in order for the NIO based components to be of any use on the client side. I am planning to start working on the NIO connection management code immediately after the alpha3 release. > Should > there be some sort of more intricate tie-in between the request & the > response it generates? (And, the built-in question, how can > handleResponse trigger another submitRequest to be called.) > IOControl#requestOutput is your friend Hope this helps Oleg > Thanks! > > Sam > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
