Author: Alexander Barkov Email: [EMAIL PROTECTED] Message: > I have the following problem, > > I am indexing *.htm, *.html files (by using the Disallow > statement) but it happens that some of thoses *.htm* files, are in fact binaries >files, (they have a wrong extension.) > > In that case, indexer stays on the file forever and doesn't go on indexing. > > Did any body encounter the same problem ?
I think that the reason is big size of those files. And probably indexer doesn't stay forever, it is just downloading for a long time. I think this is a good item for TODO: to make it possible to stop downloading of content after headers downloading, as well as to add a kind of automatic content-type checking/guessing. Reply: <http://www.mnogosearch.org/board/message.php?id=4349> ___________________________________________ If you want to unsubscribe send "unsubscribe general" to [EMAIL PROTECTED]
