On Mon, Jun 30, 2008 at 8:30 AM, Andreas Kostyrka <[EMAIL PROTECTED]> wrote: > Plus it seems to be deterministic, it always stop at 3 reduce parts > not finishing, although I haven't yet checked if they are always the same > errors or not.
I've been struggling through getting my streaming tasks (map only, no reduce) to run across large Nutch crawls. I'd been having deterministic failures as well. It's worth checking your streaming job against input data (maybe on a local workstation) to see that it doesn't blow up on some of the input. It turns out my Ruby/Hpricot XML parsers were having a hard time swallowing large binary files (surprise) and as a result, some map tasks would always die in the same place. I got my data to test locally but running streaming jar with cat as it's mapper, and then copying the results to my workstation, and piping them into my script. I haven't tried using cat as a reducer, but it should yield output files suitable for running your streaming reducers over, in an instrumented environment. -- Chris Anderson http://jchris.mfdz.com
