Nutch Newbie wrote:
Well, I would like to agree with Piotr here but current development
i.e. 0.8
version and onwards single machine nutch install is not optimal there
are various
hadoop related issue example
http://issues.apache.org/jira/browse/HADOOP-206
Is it really still a valid issue? I'm pretty sure this was already
fixed, or perhaps it was a matter of putting hard limits in
hadoop-site.xml (which overrides even job.xml values).
The problem regarding 0.8 being slow on single machine is nothing new
just search the
mailing list you will find many example for it. 0.8 was released
earlier this year and the
problem is still not solved so I am sorry to be negative but I am just
stating facts.
What Nutch needs at this moment is more developers and contributors.
This and similar issues might be solved by directly addressing each
problem, if we had human resources to do so. As it is now, there are few
active Nutch developers at the moment, and issues are being addressed
slower than we would wish it.
(BTW, Chris Mattmann will be joining the committers group, so you can
expect some improvements in this regard).
But what Piotr stated is that use cases such as yours _are_ important to
us, and this problem will be fixed sooner or later, whenever we have
free resources to do it. If you can help us with debugging and testing,
and providing patches, this process will be much quicker.
I suspect that we (Nutch community) are the only serious user of Hadoop
in local mode - most development efforts in Hadoop project are geared
towards supporting massive clusters and not single machines. So, I would
say it's up to us - the Nutch community - to provide sufficient feedback
to Hadoop to have such issues addressed.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
