Hello Martin, -----Original message----- > From:Martin Louis <[email protected]> > Sent: Mon 10-Sep-2012 16:41 > To: [email protected] > Subject: Help needed on Large scale single domain crawling ( Multiple country > / Multilanguage / user type ) CGI urls > > Hi Guys > > I am a JAVA engineer, trying to set up an environment with all the features > of GSA and more to address large website needs; > > My website > > Works mostly on CGI commands to redirect to pages > (like, ?cmd=_services-page ) > > Multiple counties ( means; they are different for different counties as > products we sell to different countries are different ); reachable by sub > URL: *mydomain.com/<country-code>/ *. > > Supports multiple local languages for each country. > > Each country can have users having multiple types of accounts ( we > support 2-3 types of users in each country based on the service level like > "free user" / "premium user" ) and the content for them will vary. > > *What will be the best approach to crawl this website for a good "site wide > search" experience both for logged -in and out users with relevant content.* > > Below are the questions with me > > 1. If i keep my *seed to be "mydomain.com"* and initiate a crawl on entire > site > >Q. How can i capture "/<country-code>/" as a field in NUTCH ) during > crawl ?
Depends on where the country-code is located, is it a HTTP element? If so, you must create a custom HTML parse filter and look for it in the DOM. Is is part of the URL? Then you can still do it with an HTML parse filter or indexing filter as they both have access to the URL and you can look it up. > >Q. How can i crawl language specific pages and index it > - Same CGI command ( like : ?cmd=_login-run ) is used for all > languages in a country > - Language flip done by setting a cookie in the website This is not going to work. The URL must be unique, see below. > > 3. My website support different types of accounts and the content can be > different for each type of account for same CGI ?cmd > > Q. How to group based on account types used to crawl. Very tricky. You must make sure the URL's are not identical. Different content for the same URL will not work in Nutch because the URL is the key in all of Nutch' databases. You can get different content for the same URL by sending different HTTP headers but in Nutch' database you will just overwrite the `other content` for the URL. > > 4. How can i do a post ( form authentication ), i know i can hack HTTP > connection, but above grouping of crawl based on authentication is blocking > me. Indeed, hack into the HTTP protocol plugin you're using. Nutch cannot do this by default. > > > > Thanks in advance, for any of your valuable suggestion to my problem > > -- > - Martin >
