Hi, I am using apache-nutch 1.4 and it is crawling perfectly. But i have got some issues in crawling some sites. For testing my crawling, I took http://www.jabong.com http://www.jabong.com I found out it is able to crawl categories but could not crawl pages.
For example look at this:- http://www.jabong.com/men/shoes/mens-sports-shoes/ -----> (Page1) Now nutch does not crawl the pages present inside this page.. URL of one of the product is:- http://www.jabong.com/Sports-White-Tennis-Shoes-2773.html -------> (Prod1) After some research, I got to know the structure of this site is: 1. Home dir contains all the product pages. If you see the source of page(Page1), it contains link to Prod1 which is actually in the home directory. So may be this is the reason it is not crawling product pages. Can some body please tell me how to solve this and make nutch to crawl such pages too. -- View this message in context: http://lucene.472066.n3.nabble.com/Nutch-not-crawling-jabong-tp3857630p3857630.html Sent from the Nutch - User mailing list archive at Nabble.com.
