Hi All Looks like DB_GONE flag is set for pages that are 404 or for pages where fetch failed for 3 or more times.
We are looking for a way to detect pages that are truly 404 or 301, to remove them from our index. Our initial plan was to use DB_GONE flag to remove from index, but later we found that some valid pages (when we curl them we get 200) are also marked as DB_GONE. Any suggestions would be appreciated. Thanks Srini
