[jira] [Resolved] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-279. --- Resolution: Fixed I just merged MR-279 branch into mapreduce trunk (finally :) ). It's been a gigantic effort. Thanks to each and everyone who made contributions, large and small! I am closing this ticket as resolved. Map-Reduce 2.0 -- Key: MAPREDUCE-279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Arun C Murthy Fix For: 0.23.0 Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, yarn-state-machine.job.dot, yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, yarn-state-machine.task.png Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCEMENT] Initiating merge of MR-279 into trunk
It is done, MR-279 branch is successfully merged into trunk! All clear for future commits. Fire up your favourite editors and IDEs to start hacking on trunk now. Thanks, +Vinod On Thu, Aug 18, 2011 at 10:56 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: With the vote passing (http://markmail.org/thread/zadfldkko2hfr3ri), I am now initiating the merge of MR-279 branch into trunk. Requesting everyone to *hold off* your commits to mapreduce trunk in the mean-while. I'll send out a note once the merge is finished. Thanks, +Vinod
Notes for working on mapreduce trunk after the MR-279 merge.
MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod
Hadoop-Mapreduce-trunk - Build # 758 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/758/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by timer Building remotely on hadoop7 Location 'http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce' does not exist One or more repository locations do not exist anymore for Hadoop-Mapreduce-trunk, project will be disabled. [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Awesome Vinod, thanks much for these instructions! Perhaps these may be recorded on the frequented wiki HowToContribute pages as well? :) On Thu, Aug 18, 2011 at 6:19 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod -- Harsh J
Re: Hadoop-Mapreduce-trunk - Build # 758 - Still Failing
Giri, can you please help with this? Apache Thanks, +Vinod On Thu, Aug 18, 2011 at 6:32 PM, Apache Jenkins Server jenk...@builds.apache.org wrote: See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/758/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by timer Building remotely on hadoop7 Location 'http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce' does not exist One or more repository locations do not exist anymore for Hadoop-Mapreduce-trunk, project will be disabled. [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Hadoop-Mapreduce-trunk - Build # 758 - Still Failing
Giri, can you please help with this? Thanks, +Vinod On Thu, Aug 18, 2011 at 6:32 PM, Apache Jenkins Server jenk...@builds.apache.org wrote: See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/758/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by timer Building remotely on hadoop7 Location 'http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce' does not exist One or more repository locations do not exist anymore for Hadoop-Mapreduce-trunk, project will be disabled. [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Notes for working on mapreduce trunk after the MR-279 merge.
I just built and brought up mrv2 on a single node cluster. I was successfully able to run wordcount from the mapreduce-examples after adding the following config change to my mapred-site.xml. Without this change it was still trying to contact the jobtracker. property name mapreduce.framework.name/name valueyarn/value /property I'll file a jira and update the INSTALL doc. Tom On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod
[jira] [Created] (MAPREDUCE-2854) update INSTALL with config necessary run mapred on yarn
update INSTALL with config necessary run mapred on yarn --- Key: MAPREDUCE-2854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2854 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Thomas Graves Assignee: Thomas Graves The following config is needed to run mapreduce on yarn framework. Document it in the INSTALL doc. property name mapreduce.framework.name/name valueyarn/value /property The INSTALL doc also still references the old 22 mapred examples jar. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Good idea. Created http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge Please feel free to edit it. Once it stabilises, we can move the content over to HowToContribute and/or something like HowToBuildAndInstall. Thanks, +Vinod On Thu, Aug 18, 2011 at 7:29 PM, Harsh J ha...@cloudera.com wrote: Awesome Vinod, thanks much for these instructions! Perhaps these may be recorded on the frequented wiki HowToContribute pages as well? :) On Thu, Aug 18, 2011 at 6:19 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod -- Harsh J
Re: Notes for working on mapreduce trunk after the MR-279 merge.
It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase? Thanks, Bobby Evans On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod
Re: Notes for working on mapreduce trunk after the MR-279 merge.
yes.. I also observed this. Looks like the mirrors are setup to use the hardcoded path to trunk/mapreduce which changed now to trunk/hadoop-mapreduce. Will see and get the right people to fix it. Thanks, +Vinod On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans ev...@yahoo-inc.com wrote: It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase? Thanks, Bobby Evans On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod
Re: Notes for working on mapreduce trunk after the MR-279 merge.
I'm kicking the git mirror now... for some reason, on merges, it often takes several commits before it fully syncs up. I have access to manually poke it into going. -Todd On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: yes.. I also observed this. Looks like the mirrors are setup to use the hardcoded path to trunk/mapreduce which changed now to trunk/hadoop-mapreduce. Will see and get the right people to fix it. Thanks, +Vinod On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans ev...@yahoo-inc.com wrote: It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase? Thanks, Bobby Evans On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod -- Todd Lipcon Software Engineer, Cloudera
Re: Notes for working on mapreduce trunk after the MR-279 merge.
It's working now. I did a pull which shows the following commits: commit cec7fc69876fb6bd50612c9282299ac94af6ab70 Author: Vinod Kumar Vavilapalli vino...@apache.org Date: Thu Aug 18 14:10:12 2011 + Updating CHANGES.txt to reflect fixes from MR-279 branch that came into trunk when merging the two of them. git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1159232 13f79535-47bb-0310-9956-ffa450edef68 commit 46702790df8d61c18ba8ace768cc7bc6b7318c0a Author: Vinod Kumar Vavilapalli vino...@apache.org Date: Thu Aug 18 11:07:10 2011 + MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey Naisbitt, Josh Wills, Jo git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1159166 13f79535-47bb-0310-9956-ffa450edef68 Thanks, +Vinod On Thu, Aug 18, 2011 at 9:24 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: yes.. I also observed this. Looks like the mirrors are setup to use the hardcoded path to trunk/mapreduce which changed now to trunk/hadoop-mapreduce. Will see and get the right people to fix it. Thanks, +Vinod On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans ev...@yahoo-inc.com wrote: It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase? Thanks, Bobby Evans On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Thanks Todd! On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon t...@cloudera.com wrote: I'm kicking the git mirror now... for some reason, on merges, it often takes several commits before it fully syncs up. I have access to manually poke it into going. -Todd On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: yes.. I also observed this. Looks like the mirrors are setup to use the hardcoded path to trunk/mapreduce which changed now to trunk/hadoop-mapreduce. Will see and get the right people to fix it. Thanks, +Vinod On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans ev...@yahoo-inc.com wrote: It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase? Thanks, Bobby Evans On 8/18/11 7:49 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce. You can get all the help that is needed from the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL. Reproducing some of those contents here for the short-term lookup. Checking out source code svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce -- Directory structure -- trunk/ - hadoop-mapreduce ( was mapreduce before) trunk/hadoop-mapreduce - Classic code. JT/TT reside here - build.xml - src trunk/hadoop-mapreduce/ - New code related to yarn reside here. - assembly - pom.xml - hadoop-mr-client - hadoop-yarn - Yarn APIs, libraries, and server code -- hadoop-yarn-api -- hadoop-yarn-common -- hadoop-yarn-server - Server code, ResourceManager, NodeManager, server libraries and tests. --- hadoop-yarn-server-common --- hadoop-yarn-server-nodemanager --- hadoop-yarn-server-resourcemanager --- hadoop-yarn-server-tests - hadoop-mr-client - MapReduce server and client code -- hadoop-mapreduce-client-app -- hadoop-mapreduce-client-core -- hadoop-mapreduce-client-jobclient -- hadoop-mapreduce-client-common -- hadoop-mapreduce-client-hs -- hadoop-mapreduce-client-shuffle --- Building --- Building yarn code and install into the local maven cache. - mvn clean install - In case you want to skip the tests run: mvn clean install -DskipTests Building classic code once yarn code is built. - ant veryclean jar jar-test -Dresolvers=internal -- Eclipse --- 1) For hacking on the new yarn+MR code in eclipse, you should run mvn eclipse:eclipse and then import the checked out source root as a maven project. 2) For developing on classic JT/TT code, running ant eclipse and importing as java project should continue to work. Hope that helps. If you run into issues, please send an email or create a JIRA issue. Thanks, +Vinod -- Todd Lipcon Software Engineer, Cloudera
[jira] [Created] (MAPREDUCE-2855) ResourceBundle lookup during counter name resolution takes a lot of time
ResourceBundle lookup during counter name resolution takes a lot of time Key: MAPREDUCE-2855 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Building MR2
Hey gang, I'm running mvn clean install from the hadoop-mapreduce directory on an svn co of trunk (hadoop-common) and get the following: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project hadoop-yarn-api: Command execution failed. Process exited with an error: 1(Exit value: 1) - [Help 1] I'm using mvn version 3.0.3. Thanks, Eli
Re: Notes for working on mapreduce trunk after the MR-279 merge.
On Thu, Aug 18, 2011 at 8:46 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Good idea. Created http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge Mind updating http://wiki.apache.org/hadoop/HowToContribute with the new MR instructions from this page? (would be good to just have one set of instructions on how to build hadoop). Thanks, Eli
Re: Building MR2
On Thu, Aug 18, 2011 at 10:29 AM, Mahadev Konar maha...@hortonworks.com wrote: Eli, I think its protoc. Do you have protoc installed? Installing protoc (and adding /usr/local/lib to my ld lib path) got it working. If we don't check in the generated sources please update the HowToContribute wiki with this dep (or make the build error nicer), otherwise we'll get lots of queries on the list. Thanks, Eli mahadev On Thu, Aug 18, 2011 at 10:28 AM, Eli Collins e...@cloudera.com wrote: Hey gang, I'm running mvn clean install from the hadoop-mapreduce directory on an svn co of trunk (hadoop-common) and get the following: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project hadoop-yarn-api: Command execution failed. Process exited with an error: 1(Exit value: 1) - [Help 1] I'm using mvn version 3.0.3. Thanks, Eli
Re: Mavenizing the HDFS build
Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Re: Mavenizing the HDFS build
+1, thanks for helping out by waiting on MR-279 - much appreciated! Arun On Aug 18, 2011, at 11:30 AM, Tom White wrote: Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Re: Building MR2
IMO, committing generated code is not a good idea. I'd rather put the burden on developers of installing protoc (they have to do it for Java, Maven, Forrest, and all the autoconf stuff if compiling native). I would document protoc as required tool. Thxs. Alejandro On Thu, Aug 18, 2011 at 10:50 AM, Eli Collins e...@cloudera.com wrote: On Thu, Aug 18, 2011 at 10:29 AM, Mahadev Konar maha...@hortonworks.com wrote: Eli, I think its protoc. Do you have protoc installed? Installing protoc (and adding /usr/local/lib to my ld lib path) got it working. If we don't check in the generated sources please update the HowToContribute wiki with this dep (or make the build error nicer), otherwise we'll get lots of queries on the list. Thanks, Eli mahadev On Thu, Aug 18, 2011 at 10:28 AM, Eli Collins e...@cloudera.com wrote: Hey gang, I'm running mvn clean install from the hadoop-mapreduce directory on an svn co of trunk (hadoop-common) and get the following: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project hadoop-yarn-api: Command execution failed. Process exited with an error: 1(Exit value: 1) - [Help 1] I'm using mvn version 3.0.3. Thanks, Eli
Re: Building MR2
On Thu, Aug 18, 2011 at 11:39 AM, Alejandro Abdelnur t...@cloudera.com wrote: IMO, committing generated code is not a good idea. I'd rather put the burden on developers of installing protoc (they have to do it for Java, Maven, Forrest, and all the autoconf stuff if compiling native). I would document protoc as required tool. Alejandro - is there an easy way we can make maven spit out a nicer error message (eg check for protoc on the path and print a message if it's not present)? (Thank you Mahadev btw for pointing out the fix!) Thanks, Eli
Re: Building MR2
protoc is invoked from an antrun plugin configuration. we could check using ant tasks that protoc is avail, and fail with a message if not. Thxs. Alejandro On Thu, Aug 18, 2011 at 11:44 AM, Eli Collins e...@cloudera.com wrote: On Thu, Aug 18, 2011 at 11:39 AM, Alejandro Abdelnur t...@cloudera.com wrote: IMO, committing generated code is not a good idea. I'd rather put the burden on developers of installing protoc (they have to do it for Java, Maven, Forrest, and all the autoconf stuff if compiling native). I would document protoc as required tool. Alejandro - is there an easy way we can make maven spit out a nicer error message (eg check for protoc on the path and print a message if it's not present)? (Thank you Mahadev btw for pointing out the fix!) Thanks, Eli
Re: Picking up local common changes in mr
populated into my local m2 repo but it is not picked up by mvn
test-compile -Dresolvers=internal
mvn doenst understand -Dresolvers=internal. (this is an ant/ivy configuration).
by default maven resolves artifacts from the local cache and if not
available then resolves from the mvn repository.
you could try with ant veryclean
On Thu, Aug 18, 2011 at 3:42 PM, Eli Collins e...@cloudera.com wrote:
Thanks Giri. That doesn't seem to work from me. Eg if I make a change
to FileUtil in hadoop-common then mvn-install -DskipTests, it gets
populated into my local m2 repo but it is not picked up by mvn
test-compile -Dresolvers=internal in hadoop-mapreduce (ie per below I
can't introduce a new method that's used in a test). Does this work
for you or am I doing something wrong?
Thanks,
Eli
hadoop-mapreduce (trunk)$ git diff
diff --git hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
hadoop
index 8e7aa30..0471caf 100644
--- hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
+++ hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
+ public static boolean fullyDelete2(File dir) { }
+
diff --git
hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apa
index 454ef2c..d029529 100644
---
hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
+++
hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
@@ -1,3 +1,4 @@
} finally {
FileUtil.fullyDelete(procfsRootDir);
+ FileUtil.fullyDelete2(procfsRootDir);
}
}
On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan
gkesa...@hortonworks.com wrote:
Hello,
Its the same -Dresolvers=internal for the ant build system; For the
maven/yarn build system as long as you have the latest common jar in
the m2 cache its going to resolve common from the maven cache. If not
from the apache maven repo. You can force the builds to use the cache
by adding -o option. (offline builds)
Thanks,
Giri
On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins e...@cloudera.com wrote:
Hey gang,
What's the new equivalent of resolvers=true in the new MR build? ie
how do you get a a local common change to get picked up by mr?
Thanks,
Eli
Re: Picking up local common changes in mr
This is handled by maven reactor. When your run Maven in a multimodule project (like we have), all modules that are part of the build (from the dir where you are) down are used for the build/test/packaging, all modules that are not part of the build are picked up from .m2/repo. For example cd trunk/hadoop-mapreduce;mvn compile uses hadoop-common hadoop-hdfs from m2/repo cd trunk;mvn compile uses hadoop-common, hadoop-hdfs, hadoop-mapreduce from the build. HTH Thxs. Alejandro On Thu, Aug 18, 2011 at 4:35 PM, Matt Foley mfo...@hortonworks.com wrote: Since we put all the effort into un-splitting the components, shouldn't we have a switch that causes, eg, the MAPREDUCE build to pick up artifacts from COMMON and HDFS builds in specified sibling directories, without using m2 as an intermediary? Of course it should respect dependencies (via maven) so that if HDFS source has been modified, the HDFS artifacts will also be rebuilt before MAPREDUCE uses them :-) --Matt On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan gkesa...@hortonworks.com wrote: Hello, Its the same -Dresolvers=internal for the ant build system; For the maven/yarn build system as long as you have the latest common jar in the m2 cache its going to resolve common from the maven cache. If not from the apache maven repo. You can force the builds to use the cache by adding -o option. (offline builds) Thanks, Giri On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins e...@cloudera.com wrote: Hey gang, What's the new equivalent of resolvers=true in the new MR build? ie how do you get a a local common change to get picked up by mr? Thanks, Eli
[jira] [Created] (MAPREDUCE-2859) mapreduce trunk is broke with eclipse plugin contrib
mapreduce trunk is broke with eclipse plugin contrib
Key: MAPREDUCE-2859
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2859
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Giridharan Kesavan
ant compile with eclipse home fails mapreduce trunk builds.
$ANT_HOME/bin/ant -Dversion=${VERSION} -Declipse.home=$ECLIPSE_HOME compile
compile:
[echo] contrib: eclipse-plugin
[javac] Compiling 45 source files to
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/contrib/eclipse-plugin/classes
[javac]
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopServer.java:39:
cannot find symbol
[javac] symbol : class JobClient
[javac] location: package org.apache.hadoop.mapred
[javac] import org.apache.hadoop.mapred.JobClient;
[javac]^
-
[javac] JobConf conf = new JobConf(location.getConfiguration());
[javac]^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 49 errors
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build.xml:451:
The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/build.xml:30:
The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/eclipse-plugin/build.xml:62:
Compile failed; see the compiler error output for details.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hadoop-Mapreduce-trunk - Build # 758 - Still Failing
eclipse plugin requires a fix. https://issues.apache.org/jira/browse/MAPREDUCE-2859 On Thu, Aug 18, 2011 at 7:09 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Giri, can you please help with this? Thanks, +Vinod On Thu, Aug 18, 2011 at 6:32 PM, Apache Jenkins Server jenk...@builds.apache.org wrote: See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/758/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by timer Building remotely on hadoop7 Location 'http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce' does not exist One or more repository locations do not exist anymore for Hadoop-Mapreduce-trunk, project will be disabled. [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-2860) Fix log4j logging in the maven test cases.
Fix log4j logging in the maven test cases. -- Key: MAPREDUCE-2860 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2860 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.0 At present the logging in the new test cases is broken because surefire isnt able to find the log4j properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
