Hi Eguzki, I wouldn't say the size of the list fitting into RAM would be the scalability bottleneck. If you're doing a full cartesian join of your users against a larger table, the fact that you're doing the full cartesian join is going to be the bottleneck first :)
-Todd On Wed, Dec 16, 2009 at 9:24 AM, Eguzki Astiz Lezaun <[email protected]> wrote: > Thanks Todd, > > That was my plan-B or workaround. Anyway, I am happy to see there is no > straight way to do so I could miss. > > The "small" list is a list of userId (dim table), so I can assume it as > "small" but that can be a limitation in the scalability of our system. I > will test the upper limits. > > Thanks a lot. > > Eguzki > > Todd Lipcon escribió: > > Hi Eguzki, >> >> Is one of the tables vastly smaller than the other? If one is small enough >> to fit in RAM, you can do this like so: >> >> 1. Add the small file to the DistributedCache >> 2. In the configure() method of the mapper, read the entire file into an >> ArrayList or somesuch in RAM >> 3. Set the input path of the MR job to be the large file. Use no reduces >> 4. In the map function, simply iterate over the ArrayList and output each >> pair. >> >> If the small file doesn't fit in RAM, you could split it into chunks >> first, >> and then run one MR job per chunk. >> Assumedly, though, one of the two smiles is small - if they're both big >> you're going to have a very very big output! >> >> -Todd >> >> On Wed, Dec 16, 2009 at 5:35 AM, Eguzki Astiz Lezaun <[email protected]> >> wrote: >> >> >> >>> Hi, >>> >>> First, I would like to apologise if this question has been asked before >>> (I >>> am quite sure it has been) and I would appreciate very much if someone >>> replies with a link to the answer. >>> >>> My question is quite simple. >>> >>> I have to files or datasets having a list of integers. >>> >>> example: >>> dataset A: (a,b,c) >>> dataset B: (d,e,f) >>> >>> I would like to design a map-reduce job to have at the ouput: >>> >>> (a,d) >>> (a,e) >>> (a,f) >>> (b,d) >>> (b,e) >>> (b,f) >>> (c,d) >>> (c,e) >>> (c,f) >>> >>> I guess this is a typical cartessian product of two datasets. >>> >>> I found ways to do joins using map-reduce, but a common key is required >>> on >>> both dataset. This is not the case. >>> >>> Any clue how to do this? >>> >>> Thanks in advance. >>> -- >>> Eguzki Astiz Lezaun >>> Technology and Architecture Strategy >>> C\ VIA AUGUSTA, 177 Tel: +34 93 36 53179 >>> 08021 BARCELONA www.tid.es >>> >>> Telefónica Investigación y Desarrollo >>> EKO Do you need to print it? We protect the environment. >>> >>> >>> >>> >> . >> >> >> > > -- > Eguzki Astiz Lezaun > Technology and Architecture Strategy > C\ VIA AUGUSTA, 177 Tel: +34 93 36 53179 > 08021 BARCELONA www.tid.es > > Telefónica Investigación y Desarrollo > EKO Do you need to print it? We protect the environment. > >
