Re: Optimize Nutch Indexing Speed

2017-06-15 Thread lewis john mcgibbney
Hi Dennis, On Thu, Jun 15, 2017 at 1:41 AM, <user-digest-h...@nutch.apache.org> wrote: > > From: Dennis A <dennis.aumil...@gmail.com> > To: user@nutch.apache.org > Cc: > Bcc: > Date: Wed, 14 Jun 2017 20:45:35 +0200 > Subject: Re: Optimize Nutch Indexing Speed

Re: Optimize Nutch Indexing Speed

2017-06-14 Thread Dennis A
l.com> > > To: user@nutch.apache.org > > Cc: > > Bcc: > > Date: Fri, 9 Jun 2017 09:59:05 +0200 > > Subject: Optimize Nutch Indexing Speed > > Hello, > > I have recently configured my Nutch crawler to index a whole domain, with > > an estimated num

Re: Optimize Nutch Indexing Speed

2017-06-14 Thread lewis john mcgibbney
Hi Dennis, On Sun, Jun 11, 2017 at 2:45 AM, <user-digest-h...@nutch.apache.org> wrote: > > From: Dennis A <dennis.aumil...@gmail.com> > To: user@nutch.apache.org > Cc: > Bcc: > Date: Fri, 9 Jun 2017 09:59:05 +0200 > Subject: Optimize Nutch Indexing Speed > He

Optimize Nutch Indexing Speed

2017-06-09 Thread Dennis A
Hello, I have recently configured my Nutch crawler to index a whole domain, with an estimated number of 1.5M-3M documents. For this purpose, I wanted to use Nutch 1.13 and Solr 4.10.4 to build a search index over these documents. The compute server is a 16 core Xeon Server with 128GB RAM. While