Hi, After searching google for crawler this is what i understand.
A crawler extracts all links of the site untill no new links are found. It keeps the links either in the database or in a file. By comparing the links and the 'keyword' of search the search engine prints the links. For example, if the link contains, 'ABC' and the search key is 'ABC' then that link will be printed. I have a members page which lists all the members of the site. It contains the name, age, SPAM, etc of the members. Since it contains more than 1000 members i am listing 10 members per page. So the link will be <a href='members.php?page_id=1'> Next </a>. The page_id will be 2..3... and goes on. If a member whose name 'ABC' will be listed in page 10 of the members page. i.e <a href='members.php?page_id=10'> will contain member 'ABC' details, and i give a search keyword 'ABC' how the crawler will get 'ABC' when it crawls the members page. Thanks
