All,
I have a MR program that I feed in a list of IDs and it generates the unique
comparison set as a result. Example: if I have a list {1,2,3,4,5} then the
resulting output would be {2x1, 3x2, 3x1, 4x3, 4x2, 4x1, 5x4, 5x3, 5x2, 5x1} or
(n^2-n)/2 number of comparisons. My code works just fine on smaller scaled sets
(I can verify less than 1000 fairly easily) but fails when I try to push the
set to 10-20k IDs which is annoying when the end goal is 1-10 million.
The flow of the program is:
1) Partition the IDs evenly, based on amount of output per value, into
a set of keys equal to the number of reduce slots we currently have
2) Use the distributed cache to push the ID file out to the various
reducers
3) In the setup of the reducer, populate an int array with the values
from the ID file in distributed cache
4) Output a comparison only if the current ID from the values iterator
is greater than the current iterator through the int array
I realize that this could be done many other ways but this will be part of an
oozie workflow so it made sense to just do it in MR for now. My issue is that
when I try the larger sized ID files it only outputs part of resulting data set
and there are no errors to be found. Part of me thinks that I need to tweak
some site configuration properties, due to the size of data that is spilling to
disk, but after scanning through all 3 sites I am having issues pin pointing
anything I think could be causing this. I moved from reading the file from HDFS
to using the distributed cache for the join read thinking that might solve my
problem but there seems to be something else I am overlooking.
Any advice is greatly appreciated!
Matt
This e-mail message may contain privileged and/or confidential information, and
is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please
notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of
this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring,
reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking
for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage
caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control
laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and
sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this
information you are obligated to comply with all
applicable U.S. export laws and regulations.
