Thanks Avery! I learned a ton doing it. As far as I could tell there were some claims on the web about YARN implementations in the works that were dated when I got to them. The only real examples I found were from hadoop-common, and were bare-bones-minimum (dist. shell) or overly complex (MRv2!) as a YARN reference. The other is Spark/Shark which is wonderful, but Scala end-to-end so not an ideal example app for all new YARNers. So it is likely we are the first Java project outside of Hadoop itself that is not a trivial application of YARN.
It was hard for me during the writing of the YARN-specific parts not to get crazy ideas. There are so many options it gives us that were not possible on MRv1. I am excited to see what sort of ideas the community will cook up to take advantage of YARN as we go forward. Anyway, I'm getting set up on a new machine and will run some more data through it and begin to strategize about a few more things I'd like to do in the short term as I have time. But the cool part is, at this point, I think its easy enough to follow that folks can jump in from here and extend it with no trouble. Anyone that wants too, feel free to bug me with questions along the way. There are also a few odd gotchas that I will document on the wiki soon. On Wed, Apr 3, 2013 at 4:25 AM, Hudson (JIRA) <[email protected]> wrote: > > [ > https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620721#comment-13620721] > > Hudson commented on GIRAPH-13: > ------------------------------ > > Integrated in Giraph-trunk-Commit #862 (See [ > https://builds.apache.org/job/Giraph-trunk-Commit/862/]) > GIRAPH-13: Port Giraph to YARN (Revision > b2dff2751d8d3d768f788b39089688c18f6c1750) > > Result = SUCCESS > ereisman : > http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=b2dff2751d8d3d768f788b39089688c18f6c1750 > Files : > * giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnClient.java > * giraph-core/src/main/java/org/apache/giraph/yarn/package-info.java > * pom.xml > * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java > * giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java > * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java > * giraph-core/src/test/resources/capacity-scheduler.xml > * giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java > * giraph-core/pom.xml > * giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java > * giraph-core/src/test/java/org/apache/giraph/yarn/TestYarnJob.java > * > giraph-core/src/main/java/org/apache/giraph/yarn/GiraphApplicationMaster.java > * giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnTask.java > * giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java > * giraph-core/src/main/java/org/apache/giraph/yarn/YarnUtils.java > * CHANGELOG > * giraph-examples/pom.xml > * checkstyle.xml > * giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java > * giraph-core/src/main/java/org/apache/giraph/bsp/BspInputFormat.java > > > > Port Giraph to YARN > > ------------------- > > > > Key: GIRAPH-13 > > URL: https://issues.apache.org/jira/browse/GIRAPH-13 > > Project: Giraph > > Issue Type: New Feature > > Reporter: Jakob Homan > > Assignee: Eli Reisman > > Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, > GIRAPH-13-3.patch, GIRAPH-13-4.patch, GIRAPH-13-5.patch, GIRAPH-13-6.patch, > GIRAPH-13-7.patch, GIRAPH-13-8.patch, GIRAPH-13-9.patch, > GIRAPH-13-9-r1.patch, GIRAPH-13-9-r2.patch, GIRAPH-13-9-r3.patch, > GIRAPH-13-9-r4.patch, GIRAPH-13-9-r5.patch, GIRAPH-13-9-r6.patch > > > > > > Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the > Hadoop trunk, we should think about what it would take to separate out the > graph processing bits of Giraph from the MR1-specific code so as to take > advantage of the less-MR centric aspects of YARN, while still supporting > both over the medium term. > > Review Board link (ready for review now): > https://reviews.apache.org/r/9811/ > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >
