[
https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209614#comment-13209614
]
Alejandro Abdelnur commented on MAPREDUCE-3868:
-----------------------------------------------
Ram,
A few tips that may be useful when doing the integration:
* If using and IDE (I use Idea) you may have to change the scope of
hadoop-minicluster to compile for the IDE to pick all necessary JARs (Idea
seems to get confused otherwise). From the command line (Maven) things work
fine with the (correct) test scope.
* You don't need to build the whole Hadoop project every time. Instead do the
following:
** Run once, from root level 'mvn install -DskipTests'
** then CD to hadoop-hdfs-project/hadoop-hdsf-raid and work from there.
** You can do Maven command or point your IDE to the POM there.
** Make sure you are using -offline with Maven command and in your IDE (else
you may pick up newer snapshot JARs/POMs from Apache that don't have the
changes this patch added)
* The new YARN base MiniCluster still has some issues, on failures you have
runaway processes (the shutdown logic is broken, a JIRA to fix that is in the
works MAPREDUCE-3634)
** What i do is a 'ps -ef | grep java' and then kill the dangling processes
* Sometimes (at least in Mac OS) MiniDFS finds some local files and things it
is locked and testcases fail because of that. If that is the case, delete those
files.
* After doing a build javadocs stuff sometimes creates a dir api at src level,
ignore that DIR (don't add it to your patch)
* the distribution layout follows the new layout, I've put the scripts in the
right places, but I have not modified the scripts to look for things in the
right places
* the full distribution must be done from root level, do 'mvn clean package
-DskipTests -Pdist', the full distribution will be in hadoop-dist/target/....
* the testcases that start MR jobs (and maybe the main code) will have to setup
the framework to YARN, you have to set in your job conf the following property:
mapreduce.framework.name=yarn (the streaming porting did this HADOOP-7590)
* in the testcases, you may have to add your classes as attached JAR, use the
JarFinder.getJar() class to get the RAID JAR (the streaming porting did this
HADOOP-7590).
* Some MiniMRCluster methods related to MR1 throw
UnsupportedOperationExceptions, you may have to find other ways of getting the
right info. For example: MiniMRCluster.getJobTrackerPort() now throws UOE, you
should use instead MiniMRCluster.createJobConf().get(JTConfig.JT_IPC_ADDRESS)
* I've modified the TestBlockFixer with the last 2 items and
TestBlockFixerBlockFixDist.testBlockFixDist() test get to start and run for
long before failing.
> Reenable Raid
> -------------
>
> Key: MAPREDUCE-3868
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/raid
> Reporter: Scott Chen
> Assignee: Ramkumar Vadali
> Attachments: MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh
>
>
> Currently Raid is outdated and not compiled. Make it compile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
