Thank you Marcus for your fast response! 1) If www.somesite.com redirects to www.somesite.com, will it fetch it at the next cycle? 2) I understand that the whole set of urls to be fetched is saved at QueueFeeder. Each thread will be assigned a number of urls to fetch equal to: (wholeSetToBeFetched) / (numberOfThreads) ?
Happy Valentine's Day! > Subject: RE: Threads > From: [email protected] > To: [email protected] > Date: Fri, 14 Feb 2014 11:45:16 +0000 > > Hi, > > They take records or (FetchItems) from the QueueFeeder. Queues are based on > domain, host or ip and a URL exists only once, so nothing collides. The > redirect will be followed in the next fetch cycle. > > Markus > > > > -----Original message----- > > From:Vangelis karv <[email protected]> > > Sent: Friday 14th February 2014 12:39 > > To: [email protected] > > Subject: Threads > > > > Hello people! > > > > Lets say we choose 20 threads to fetch. How do they cooperate? I mean, who > > tells them what pages each one of them will fetch? > > Is it possible some of them to collide or fetch the same page without them > > knowing? > > I read the code and found that if the redirect is to the same page, it will > > not follow that redirect. Any advice would be very helpful! > > > > Vangelis >
