There shouldn't be a problem with lists that size. We have done initial injections with > 5MM pages per fetchlist.
Dennis Kubes Hermann Rokicz wrote: > Hi, > > I'm planing to use nutch to crawl between 1 and 2 millionen domains. > From the documentation i guess intranet crawling would be the right > method. > > Are there known problems with intranet crawling and this size of > domainlist? > > Regards, > Hermann! ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
