On Thursday 11 December 2008 00:38, [EMAIL PROTECTED] wrote: > Author: j16sdiz > Date: 2008-12-11 00:38:31 +0000 (Thu, 11 Dec 2008) > New Revision: 24187 > > Modified: > trunk/plugins/XMLSpider/XMLSpider.java > Log: > be smart - don't fetch images, css, etc...
I didn't do this when I built the spider because I figured it wouldn't gain us
much speed, since we only ever download the top layer before finding the MIME
type and realising we don't want the data, and would lose us some
comprehensiveness. But you're probably right.
>
> Modified: trunk/plugins/XMLSpider/XMLSpider.java
> ===================================================================
> --- trunk/plugins/XMLSpider/XMLSpider.java 2008-12-10 16:49:17 UTC (rev
24186)
> +++ trunk/plugins/XMLSpider/XMLSpider.java 2008-12-11 00:38:31 UTC (rev
24187)
> @@ -199,6 +199,18 @@
> * @param uri the new uri that needs to be fetched for further indexing
> */
> public synchronized void queueURI(FreenetURI uri) {
> + String sURI = uri.toString();
> + if (sURI.endsWith(".png") ||
> + sURI.endsWith(".jpg") ||
> + sURI.endsWith(".css") ||
> + sURI.endsWith(".gif") ||
> + sURI.endsWith(".zip") ||
> + sURI.endsWith(".avi") ||
> + sURI.endsWith(".ico") ||
> + sURI.endsWith(".xpi") ||
> + sURI.endsWith(".iso"))
> + return; // be smart
> +
> if (uri.isUSK()) {
> if(uri.getSuggestedEdition() < 0)
> uri = uri.setSuggestedEdition((-1)*
> uri.getSuggestedEdition());
>
> _______________________________________________
> cvs mailing list
> [EMAIL PROTECTED]
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
>
>
pgpCmNTy6pg7e.pgp
Description: PGP signature
_______________________________________________ Devl mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
