Titus Brown wrote: > On Mon, Dec 18, 2006 at 12:33:18PM +0100, Lars Stavholm wrote: > -> Titus Brown wrote: > -> > On Sat, Dec 16, 2006 at 05:31:43PM +0100, Lars Stavholm wrote: > -> > -> Lars Stavholm wrote: > -> > -> > I'm doing a simple all-links check on a site > -> > -> > using the following simple twill script: > -> > -> > > -> > -> > go http://www.jonitec.se > -> > -> > extend_with check_links > -> > -> > check_links www\.jonitec\.se > -> > -> > > -> > -> > But for some reason, it doesn't follow the links. > -> > -> > > -> > -> > Does anyone have any idea why? > -> > -> > -> > -> Here's another one: <http://milleteknik.se>. > -> > -> Some of the links are followed, some not. > -> > -> > -> > -> Any ideas? > -> > -> /L > -> > > -> > Hi, Lars, > -> > > -> > short answer: it's not finding them! 'showlinks' isn't seeing anything > -> > on jonitec. I have no idea why; I have to dig into the mechanize code > -> > to find out. > -> > -> OK, fine so, the only thing I can add is that the site is a php > -> CMS based site (joomla.org) and that I'll leave it as is for now. > -> The links in the generated HTML look fine to me. > > Hi, Lars, > > neither tidy nor BeautifulSoup like the conditionals in the HTML on this > page; the culprit on jonitec.se appears to be this: > > === > > <!-- CorrectPNG! Module : compliance patch for microsoft browsers --> > <!--[if gte IE 5.5000]> > <!--[if lte IE 7]><script language="JavaScript" > src="http://www.jonitec.se/mambots/system/botcorrectpng/correctpng.js";></script><![endif]--> > <![endif]--> > > === > > That is, if I remove that from the page, showlinks works fine. The > <![endif]--> is specifically what's causing the problem; if you put > <!--[endif]--> link parsing works. > > Do you have any thoughts on how to deal with this? It's obviously > incorrect HTML but it shouldn't be breaking things this badly ;).
Thanks Titus! Problem solved in the sense that I fixed the parsing problem by simply making sure that correct HTML is produced following your advice and findings. I'll remember to try and make sure that I have correct HTML before running twill again. >From an overall perspective, I'd say that we're done. One should really require correct HTML to be produced. However, is there any way one could tell twill/tidy/BeautifulSoup to produce an error message when incorrect HTML is discovered? That would be handy in a testing tool like twill. As it stands now, I got a bit lost. Thanks! /Lars _______________________________________________ twill mailing list [email protected] http://lists.idyll.org/listinfo/twill
