Hi Yoni, That was a good thought, however, according to the logging output of the crawl, I see the following...
fetching http://www.visitpa.com/visitpa/details.pa?id=65851 fetching http://www.visitpa.com/visitpa/details.pa?id=246139 fetching http://www.visitpa.com/visitpa/details.pa?id=8427 There are at least 100+ of these (too many to count) so it appears that nutch is fetching these url's although the url is not unique without the query string. Building upon your thought, perhaps the other "details.pa" pages are coming from other pages being indexed, and only one "details.pa" page is being used in the sense of a crawl. That could be what is happening here and your point is correct. I appreciate your response! Phillip ----- Original Message ----- From: "Yoni Amir" <[EMAIL PROTECTED]> To: <nutch-user@lucene.apache.org> Sent: Wednesday, December 06, 2006 10:47 AM Subject: Re: page1 is crawled, but not pages in page1 The I think that in crawldb and linkdb, the actual url, without the query string, serves as primary key (i.e. a url is determined as unique just by looking at the url, without the query string). Thus, after your first page is fetched, and you run updatedb, nutch doesn't think that it needs to fetch it again because it already sees an entry for it in the database. I am also new to nutch, so I don't know if there is a solution to your problem. Yoni On Wed, 2006-12-06 at 10:05 -0500, spamsucks wrote: > My subject is a pretty good summary. I see the first "details.pa?id=123" > in > my results, but can't search or find any "details.pa?id=456" links that > are > in that 1st page that was a hit. > > Backgrounder: > I have a site that includes a lot of dynamic pages. I edited the > crawl-urlfilter.txt and added the following regex and did > a crawl (bin/nutch crawl urls -dir crawl -depth 30 -topN 30000): > > +^http://([a-z0-9]*\.)*www.visitpa.com/visitpa/details.pa\?id= > > Now the search will return hits on the dynamic details page. For example, > here is a search that returns hits on my dynamic pages. > http://prhodes.r-effects.com/nutch/search.jsp?query=sunnyledge&hitsPerPage=10〈=en > > If you look at the details.pa page that nutch had a hit on, it contains > several links of the same format ( details.pa ) > My problem is that these other detail links are not being crawled/indexed. > > I set the depth to "30" so that should not be a limiting factor. I also > set > a "topN" of 30000, because we have around 16K details.pa pages > > Any clues on how to proceed and figure out what I need to do to get Nutch > to > crawl these missing "details.pa" links > > > > > ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list Nutch-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-general
