Hi Dennis,
But it doesn't make sense since the reducers' keys are URLs and the
heartbeat cannot be sent when the reduce task is called. Since I am
truncating my http content to be less than 100K and I don't get any file,
how come reducing a single record which is a single URL and writing its
parsed data into DFS takes more than 10 min!! Even if you load the cluster
that should never happen. There should be another bug involved.
Thanks. Nima
On 10/17/06, Dennis Kubes <[EMAIL PROTECTED]> wrote:
I have seen this happen before if the box is loaded down with too many
tasks and the IO is maxed. I have also seen this happen when the regex
filters spin out. We changed our systems to use only prefix and suffix
url filters and that cleared up those types of problems for us.
Dennis
Mike Smith wrote:
> Hi,
>
>
>
> I've been running the latest trunk nutch version on a cluster of 10
> machines. Fetch mappers always finish without any problem over 4,000,000
> pages, but some reducers fails because of "Task failed to report
> status for
> 602 seconds. Killing." Once this task fails even if it gets assigned to
> another machine that fails again.
>
>
>
> I checked the reducer of the fetcher class, and it seems to be an
> identity
> reducer that gets stuck for one key and doesn't move any further. I am
> not
> storing any http contents or files, why reducer should take this long
> for a
> key which is a URL and its content is limited to 100,000 bytes.
>
>
>
> These faulty reducers do the copying and sorting (up to 66%) without any
> problem, and then they get stuck in the reduce stage.
>
>
>
> Thank. Mike
>
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
