Right, that's the concept I'd had in mind, but to me it always seem to come down to having access to two distinct vectors at the same time, and I'm not sure how you would do that. In my case, both the dimensions and the data types of the two vectors are identical, so we're talking a merged vector of floats that's simply twice as long as the original, but how to gain access to the two original vectors at the same time is beyond me.
But still, the data types I need that would do this for me are in a newer Hadoop commit, I'm just trying to figure out how to build the commit manually and integrate it to the core Hadoop .jar file. Any suggestions that would speed along either of these options are most welcome. Shannon On Tue, Aug 3, 2010 at 11:50 AM, Sean Owen <[email protected]> wrote: > What I ended up doing in this case, IIRC, is to use another phase to > convert inputs 1 and 2 into some contrived new single Writable format. > Then both sets of input are merely fed into one mapper. So I'd > literally have Writable classes that contained, inside, either a > FooWritable or BarWritable. A little ugly but not bad. > > On Mon, Aug 2, 2010 at 3:24 PM, Shannon Quinn <[email protected]> wrote: > > CompositeInputFormat implements a hadoop.mapred.join interface, whereas > > job.setInputFormatClass() is expecting a class that extends a > > hadoop.ioclass. Also, TupleWritable is in the deprecated hadoop.mapred > > package, too. > > > > Still hunting around the API for the newer equivalent; there has to be a > way > > of doing this? > > > > On Mon, Aug 2, 2010 at 6:20 PM, Jake Mannix <[email protected]> > wrote: > > > >> On Mon, Aug 2, 2010 at 3:13 PM, Shannon Quinn <[email protected]> > wrote: > >> > > >> > Excellent. Any idea what the Hadoop 0.20.2 equivalent for > >> > CompositeInputFormat is? :) > >> > > >> > >> Ah, there is that part. Hmm... it's really really annoying to not have > >> that > >> in 0.20.2. > >> > >> This is actually why I haven't migrated the distributed matrix stuff to > the > >> newest > >> Hadoop API - map-side join is pretty seriously useful sometimes. > >> > >> Does the old CompositeInputFormat work with the new API, does anyone > know? > >> > >> -jake > >> > > >
