thanks d_k for all your help! But its a production environment and all the servers in that environment are restricted by the firewall. I am pretty sure that I will not find a server that is open to the internet for running nutch in that environment. :(
On Wed, Feb 12, 2014 at 11:21 AM, d_k <[email protected]> wrote: > If you're behind a firewall then I think your best bet would be to either > open port 8983 and run nutch on a server open to the internet and have it > index documents to solr over the open port 8983 that will only accept the > required HTTP headers or better yet, if its going to be a static index and > no one else will be writing to it setup solr on the same server as nutch. > index locally and then sftp the index to your network behind the firewall. > You can probably just copy the entire solr directory and it should work. > > > On Wed, Feb 12, 2014 at 5:44 PM, A Laxmi <[email protected]> wrote: > > > Hi d_k! Yes, I am indexing them using Solr. > > Solr is also running on the same server on port 8983. I plan to perform > the > > crawl every 30 days to update the old crawled data and to crawl any new > > sites. > > > > > > On Wed, Feb 12, 2014 at 1:37 AM, d_k <[email protected]> wrote: > > > > > What are you doing with the crawled data? > > > If you index it using solr then you can open the port solr is listening > > on > > > and run nutch on a server without a firewall and have it send the > > documents > > > to the solr behind your firewall using the port you opened. > > > > > > Is it a one time crawl? How often do you plan to perform the crawl? > > > > > > > > > On Wed, Feb 12, 2014 at 3:53 AM, A Laxmi <[email protected]> > wrote: > > > > > > > Hi, > > > > > > > > I installed Nutch 2.2.1 on a server that is restricted by firewall to > > > > access internet. I tried to run my first crawl on that server, I > > started > > > > getting timedout errors and the crawl was getting hung. So, firewall > > was > > > > actually blocking my nutch crawler to crawl any site. I verified that > > > with > > > > hosting admin and they mentioned firewall does block the crawler from > > > > crawling websites. > > > > > > > > I am not sure how I go about getting nutch to crawl websites in such > a > > > > firewall restricted environment? Please suggest > > > > > > > > Thanks! > > > > > > > > > >
