Please see my reply inline. On Thu, May 8, 2008 at 12:04 PM, POIRIER David <[EMAIL PROTECTED]> wrote: > Yoav, > > You are right. With the help of the "protocol-httpclient" plugin you > will be able to use cookies when crawling. There is one thing that you > need to watch out though (quoting Susam Pal): "protocol-httpclient does > this for a single fetch cycle". > > To be honest I don't exactly know how to define a "fetch cycle". Based > on my experience it seems that every time the fetcher goes one level > deeper into a web site it starts a new cycle... or if it doesn't I loose > the cookie. It might be because of something else, but I don't think so.
Yes, that's what I meant. For a crawl at a new depth, a new fetcher process is invoked. The cookies are not saved between processes. So, everytime the crawl goes one level deeper, the cookies are lost. Regards, Susam pal > > If anybody has the answer to that, please let Yoav and I know. > > Thanks, > > David
