Sami Siren wrote: > Brian Whitman wrote: > >> On Jan 21, 2007, at 6:47 AM, Sami Siren wrote: >> >> >>>> However I cannot find from the change logs of hadoop that what the >>>> change is that is causing nutch these problems. >>>> >>> It's HADOOP-331, so i guess at least the changes/additions in map() is >>> required. >>> >> Hi, just following up here-- does this indicate that if I get a hadoop >> nightly that was patched for HADOOP-331 and have Nutch use it, the >> EOFException will go away in the latest nightlies? >> > > No, I mean that HADOOP-331 is the change that is _causing_ these, so we > need to adapt nutch code to coop with the change in sorting. > > Is there somebody that can tell me why the various utilities (like > Indexer) is doing the wrapping to ObjectWritable in InputFormat and not > in Mapper.map in the first place? Is this optimization of some kind? >
This is a legacy from the (very recent) times when you had to set a key/value class of the InputFormat in your mapred job. You don't have to do this now - it's handled transparently by InputFormat.getRecordReader().createKey() and createValue(). In fact, there's a lot of this cruft left over in Nutch. We should also use GenericWritable in most of these places, and indeed we could wrap the values in Mapper.map(). -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
