Thanks for the answer, Oleg. Here's a little background of what I'm doing: we have an in-house scripting engine to access and process a set of pages. The objective is to provide authentication and parsing to other search engines (think Britannica and other sites for students). So we mimic a user logging in and searching as many databases as needed, then we parse, group, and classify those entries.
Each thread accesses a single site for each search. The pages might or might not be on the same server. Most likely they will be, but some authentication pages take you somewhere else (server farms). The whole search should not take more than a configurable time. E.g. we might need to: 1 - post an authentication, retrieve the response, run a regex to validate that the login happened 2 - run a fetch to set the database to be searched, or to retrieve a special cookie, etc 3 - post a search, clip the result section, run a regex to extract entries, post-process those entries 4 - request a second page of results (if they are available) 5 - get an URL to logout (for one or 2 databases) The whole process has a general timeout. So for the first step, I might have 30 secs. If it takes 5 secs, I need the connection and first read timeout for the second step to be limited to 25 secs and so on. I am trying to update the timeouts before each network access (connect or read). If the connection is kept alive and I'm requesting a connection that's still open, that's ok, there won't be a timeout there anyway. But I don't have that information until I'm the very last second. As long as the 2 connections are not used simultaneously (I'm using the multithreaded conn manager). Currently, I have a fixed timeout of 20 secs for connection and 10 secs per read. It's more than I'd like to have for an average, but it's less than a few websites require. School library software is a mess :) Maybe I should start an open source project for that :/ []s Gustavo -----Original Message----- From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] Sent: December 13, 2004 3:58 PM To: HttpClient User Discussion Subject: RE: How to set timeout parameters for a method? Gustavo, (1) The only way I can think of to provide control over socket timeout during method's execution is to override GetMethod#executeMethod, store the reference to the connection object, and provide direct access to its properties. (2) The connection timeout on the method level makes no sense to me of what so ever. Most likely the connection will be kept alive between subsequent method executions, anyways. It is (almost) completely non- deterministic. There's no way to say if you get a new connection or an open connection from the pool. I may be wrong here as I do not really know what you are trying to achieve with your application but I would urge you to rethink the design. Oleg On Mon, 2004-12-13 at 16:04 -0600, Gustavo Hexsel wrote: > I can't, the values are different for each run. The reason is because I > have a "general timeout" which includes retrieving and parsing a variable > number of pages. > > []s Gustavo > > > -----Original Message----- > From: Jilles van Gurp [mailto:[EMAIL PROTECTED] > Sent: December 13, 2004 2:41 PM > To: HttpClient User Discussion > Subject: Re: How to set timeout parameters for a method? > > > Don't set the timeouts on the method but on the HttpClient instance. You > can use the setConnectionTimeout and setTimeout methods. > > Jilles van Gurp > > Gustavo Hexsel wrote: > > > Greetings, > > > > does anyone know how to set the timeout parameters (if possible, both > >socket and connection) for a particular method? I tried to make an > accessor > >to the HttpConnection, but as I suspected, it is not created until you try > >to execute the method, and by then is too late, as both connection and > first > >read already happened. > > I tried to do something like this: > > > > method.getParams().setIntParameter("http.socket.timeout", > >remainingTime); // in my logs, remaining time is about 300s > > method.getParams().setIntParameter("http.connection.timeout", > >remainingTime); > > > > Even though "connection timeout" is not supposed to be a valid parameter > >for the method. The log indicated that the parameters were set, but the > >values remained the same as the defaults set before, and I get a > >"SocketTimeout: Read timed out" after about 10s (my default param setting). > > Similarly, even if the first "read()" doesn't time out, I've set the > >connection timeouts to the remaining time (over 280s by then), but still > >times out in the default values (10s). This is the code I'm using: > > > > HttpConnectionParams connParams = > >method.getResponseConnection().getParams(); // I just provided an accessor > >here > > connParams.setConnectionTimeout(remainingTime); > > connParams.setSoTimeout(remainingTime); > > > > > > Any ideas? > > > > > > Thanks a lot, > > > > Gustavo > > > > > >--------------------------------------------------------------------- > >To unsubscribe, e-mail: [EMAIL PROTECTED] > >For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
