I recently upgraded to 0.9, and I've started encountering a problem. I began
with a single url and crawled with a depth of 10, assuming I would get every
page on my site. This same configuration worked for me in 0.8. However, I
noticed a particular url that I was especially interested in was not in the
index. So I added the url explicitly and crawled again. And it still was not in
the index. So I checked the logs, and it is being fetched. So I tried a lower
depth, and it worked. With a depth of 6, the url does appear in the index. Any
ideas on what would be causing this? I'm very confused.
Thanks,
Ann
____________________________________________________________________________________Pinpoint
customers who are looking for what you sell.
http://searchmarketing.yahoo.com/
