[Robots] Testing a Web Crawler

2004-05-26 Thread White, Norman E.
Hi Fellow Crawler Creators, I am a PhD student at George Mason University, majoring in Machine Learning and Data Mining under Dr. R. Michalski. My long range goal is to transform text into useful knowledge (a semantic net). I am also interested in the inference process (how to infer

Re: [Robots] Testing a Web Crawler

2004-05-26 Thread Klaus Johannes Rusch
White, Norman E. wrote: For example, How do I know if I am pulling in all the pages that I should? How do I know if I am correctly extracting all the links from each page? (Besides links on html pages, there are links on MS Word pages, and other types of pages, some in somewhat different