Looks pretty interesting. Does it just scrape the home page or does it go deeper?
Also is there any was to see the scraped HTML? David Neilsen | 07 834 3366 | PANmedia ® On Sat, Jun 23, 2012 at 12:48 AM, Nick <[email protected]> wrote: > Evening (or morning to be precise), > > Just wanted to share something that I've spent a little bit of time on, I > got it up the other week and finally thought I'd let other people see it. > > http://www.crawl.co.nz/ > > Basically, I'm trying to get all the NZ domains, my first "crawl" i've > come up with around a third, which isn't to bad. I've put up an index of > the crawl here, unlike search engines you can search the html here, so you > can see which sites running jquery, or some other piece of js. Can also do > ands and ors, search ips and headers. > > Have a look :) Working on getting another 20-30% on the next crawl. > > Any questions, just ask. > > -- > NZ PHP Users Group: http://groups.google.com/group/nzphpug > To post, send email to [email protected] > To unsubscribe, send email to > [email protected] -- NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [email protected]
