On Nov 26, 2008, at 13:19, Julian Reschke wrote:
So wouldn't it make sense to address the common use case so that it doesn't require the "bot" (the non-HTML UA) to parse the response body?
That seems like a bad optimization. Adding an off-the-shelf HTML parser to a bot is much easier than tuning the general crawling functionality and task-specific functionality of a bot.
-- Henri Sivonen [EMAIL PROTECTED] http://hsivonen.iki.fi/
