On 27/8/05 6:40 AM, "Bob Wyman" <[EMAIL PROTECTED]> wrote:
> I think "crawling" URI's found in <link/> tags, > <img/> tags and enclosures isn't crawling... Or... Is there something I'm > missing here? crawling <img> tags isn't a huge problem because it doesn't lead to a recursive situation. Same withh stylesheets. Mark's point with enclosures is that they tend to be big and the basic principle involved is to minimise harm, not minimise a certain behaviour know to cause harm sometimes. Following <link> tags which lead to more <feed> or <entry> documents, which can also contain more <link> tags, and thus recursive behaviour, is definitely crawling behaviour which is covered by robots.txt e.
