Re: merge mapred to trunk
I will postpone the merge of the mapred branch into trunk until I have a chance to (a) add some MapReduce documentation; and (b) implement MapReduce-based dedup. Doug Doug Cutting wrote: Currently we have three versions of nutch: trunk, 0.7 and mapred. This increases the chances for conflicts. I would thus like to merge the mapred branch into trunk soon. The soonest I could actually start this is next week. Are there any objections? Doug
Re: merge mapred to trunk
Doug Cutting wrote: Currently we have three versions of nutch: trunk, 0.7 and mapred. This increases the chances for conflicts. I would thus like to merge the mapred branch into trunk soon. The soonest I could actually start this is next week. Are there any objections? Doug +1 P.
Re: merge mapred to trunk
Currently we have three versions of nutch: trunk, 0.7 and mapred. This increases the chances for conflicts. I would thus like to merge the mapred branch into trunk soon. The soonest I could actually start this is next week. Are there any objections? I, too, am looking forward to this, but I am wondering what that will do to Kelvin Tan's recent contribution, especially since I saw that both MapReduce and Kelvin's code change how FetchListEntry works. If merging mapred to trunk means losing Kelvin's changes, then I suggest one of Nutch developers evaluates Kelvin's modifications and, if they are good, commits them to trunk, and then makes the final pre-mapred release (e.g. release-0.8). Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ -- Find it. Tag it. Share it.
Re: merge mapred to trunk
[EMAIL PROTECTED] wrote: I, too, am looking forward to this, but I am wondering what that will do to Kelvin Tan's recent contribution, especially since I saw that both MapReduce and Kelvin's code change how FetchListEntry works. If merging mapred to trunk means losing Kelvin's changes, then I suggest one of Nutch developers evaluates Kelvin's modifications and, if they are good, commits them to trunk, and then makes the final pre-mapred release (e.g. release-0.8). It won't lose Kelvin's patch: it will still be a patch to 0.7. What I worry about is the alternate scenario: that Kelvin others invest a lot of effort making this work with 0.7, while the mapred-based code diverges even further. It would be best if Kelvin's patch is ported to the mapred branch sooner rather than later, then maintained there. Doug
Re: merge mapred to trunk
--- Doug Cutting [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: I, too, am looking forward to this, but I am wondering what that will do to Kelvin Tan's recent contribution, especially since I saw that both MapReduce and Kelvin's code change how FetchListEntry works. If merging mapred to trunk means losing Kelvin's changes, then I suggest one of Nutch developers evaluates Kelvin's modifications and, if they are good, commits them to trunk, and then makes the final pre-mapred release (e.g. release-0.8). It won't lose Kelvin's patch: it will still be a patch to 0.7. Ah, right, we could always make a 0.7.* release from release 0.7. What I worry about is the alternate scenario: that Kelvin others invest a lot of effort making this work with 0.7, while the mapred-based code diverges even further. It would be best if Kelvin's patch is ported to the mapred branch sooner rather than later, then maintained there. I agree. I'll actually see Kelvin in person tomorrow, so we'll see if this is something he can do. It looks like he added some much-needed functionality in his patch, so it'd good to keep it. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ -- Find it. Tag it. Share it.
Re: merge mapred to trunk
On Wed, 31 Aug 2005 14:37:54 -0700, Doug Cutting wrote: [EMAIL PROTECTED] wrote: I, too, am looking forward to this, but I am wondering what that will do to Kelvin Tan's recent contribution, especially since I saw that both MapReduce and Kelvin's code change how FetchListEntry works. If merging mapred to trunk means losing Kelvin's changes, then I suggest one of Nutch developers evaluates Kelvin's modifications and, if they are good, commits them to trunk, and then makes the final pre-mapred release (e.g. release-0.8). It won't lose Kelvin's patch: it will still be a patch to 0.7. What I worry about is the alternate scenario: that Kelvin others invest a lot of effort making this work with 0.7, while the mapred- based code diverges even further. It would be best if Kelvin's patch is ported to the mapred branch sooner rather than later, then maintained there. Doug Agreed. I have some time in the coming weeks, and will work fulltime to evolve the patch to be more compatible with Nutch especially map-red.. k
