[
https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-13:
------------------------------
Attachment: GIRAPH-13-8.patch
This version runs to completion (as in output gets written) when running
examples on pure yarn profile. If you have 2.0.3-alpha installed, build giraph
like:
{code}
mvn -Phadoop_yarn clean package
{code}
then use the giraph-examples jar with deps to run using a command line such as
this:
{code}
bin/hadoop --config etc/hadoop jar
share/hadoop/giraph/giraph-examples-0.2-SNAPSHOT-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.ConnectedComponentsVertex -w 1 -vif
org.apache.giraph.io.formats.IntIntNullIntTextInputFormat -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
/user/ereisman/graph -op /user/ereisman/output
{code}
A lot of the functionality we need (CLI opts etc) is already there, you can run
your own jars or include whatever you want in the job, and it will run on
cluster too.
It still needs a lot of cleanup, tests, and I have to get the YARN setup code
to end the job nicely (fail or success) and I need the output committed to the
right place. And I will remove the hardcoded dep on giraph-examples ;) All that
will be in the next patch.
But, this is working and will commit Giraph output to HDFS. All setStatus msgs
end up in the logs for now. You may have to CTRL-C out of the Yarn Client when
its done (for now but not for long...)
More to follow...
> Port Giraph to YARN
> -------------------
>
> Key: GIRAPH-13
> URL: https://issues.apache.org/jira/browse/GIRAPH-13
> Project: Giraph
> Issue Type: New Feature
> Reporter: Jakob Homan
> Assignee: Eli Reisman
> Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch,
> GIRAPH-13-4.patch, GIRAPH-13-5.patch, GIRAPH-13-6.patch, GIRAPH-13-7.patch,
> GIRAPH-13-8.patch
>
>
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop
> trunk, we should think about what it would take to separate out the graph
> processing bits of Giraph from the MR1-specific code so as to take advantage
> of the less-MR centric aspects of YARN, while still supporting both over the
> medium term.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
