Dear list subscribers, I'd like to crawl web sites and **only** download from them pages that are of content type "text/html". After studying WGet manual, I found that the closest thing WGet allows is to define a broad set of file extensions through "-A", and force "text/html" files to ".html" extension even if they were originally produced by some script such as ".php/.asp/...". But this way I effectively download even some non-"text/html" files, and then have to filter them according to their extension (forced ".html" vs. all the rest).
I wonder if there is a direct way to tell WGet to only download "text/html" files ? Thank you in advance, Evgeniy. P.S. In your reply, please CC: me at mailto:[EMAIL PROTECTED] -- Evgeniy Gabrilovich Ph.D. student in Computer Science Department of Computer Science, Technion - Israel Institute of Technology Technion City, Haifa 32000, Israel E-mail: [EMAIL PROTECTED] WWW: http://www.cs.technion.ac.il/~gabr Phone: (office) +972-4-8294948
