[jira] [Updated] (GIRAPH-184) Upgrade to junit4
[ https://issues.apache.org/jira/browse/GIRAPH-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated GIRAPH-184: - Attachment: GIRAPH-184-2.patch > Upgrade to junit4 > - > > Key: GIRAPH-184 > URL: https://issues.apache.org/jira/browse/GIRAPH-184 > Project: Giraph > Issue Type: Bug >Reporter: Devaraj K > Attachments: GIRAPH-184-1.patch, GIRAPH-184-2.patch, GIRAPH-184.patch > > > Presently Giraph uses JUnit 3.8.1. We can upgrade to JUnit 4 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-184) Upgrade to junit4
[ https://issues.apache.org/jira/browse/GIRAPH-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated GIRAPH-184: - Attachment: GIRAPH-184-1.patch > Upgrade to junit4 > - > > Key: GIRAPH-184 > URL: https://issues.apache.org/jira/browse/GIRAPH-184 > Project: Giraph > Issue Type: Bug >Reporter: Devaraj K > Attachments: GIRAPH-184-1.patch, GIRAPH-184.patch > > > Presently Giraph uses JUnit 3.8.1. We can upgrade to JUnit 4 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-184) Upgrade to junit4
[ https://issues.apache.org/jira/browse/GIRAPH-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated GIRAPH-184: - Attachment: GIRAPH-184.patch > Upgrade to junit4 > - > > Key: GIRAPH-184 > URL: https://issues.apache.org/jira/browse/GIRAPH-184 > Project: Giraph > Issue Type: Bug >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: GIRAPH-184.patch > > > Presently Giraph uses JUnit 3.8.1. We can upgrade to JUnit 4 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-175) Replace manual array copy to utility method call
[
https://issues.apache.org/jira/browse/GIRAPH-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated GIRAPH-175:
-
Attachment: GIRAPH-175.patch
> Replace manual array copy to utility method call
>
>
> Key: GIRAPH-175
> URL: https://issues.apache.org/jira/browse/GIRAPH-175
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Trivial
> Attachments: GIRAPH-175.patch
>
>
> {code} String[] zkJavaOptsArray = zkJavaOptsString.split(" ");
> if (zkJavaOptsArray != null) {
> for (String javaOpt : zkJavaOptsArray) {
> commandList.add(javaOpt);
> }
> }{code}
> Rather than doing the loop ourselves, Collections.addAll would be simpler
> (and faster, though that doesn't matter with such a small array). Still
> cleaner, though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-176) BasicRPCCommunications has unnecessary cast of Vertex
[
https://issues.apache.org/jira/browse/GIRAPH-176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated GIRAPH-176:
-
Attachment: GIRAPH-176.patch
> BasicRPCCommunications has unnecessary cast of Vertex
> -
>
> Key: GIRAPH-176
> URL: https://issues.apache.org/jira/browse/GIRAPH-176
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Minor
> Attachments: GIRAPH-176.patch
>
>
> BasicRPCCommunications.java, 1224:
> {code} BasicVertex vertex =
> vertexResolver.resolve(vertexIndex,
> originalVertex,
> vertexMutations,
> messages);{code}
> and then a few lines later at 1248:
> {code}partition.putVertex((BasicVertex) vertex);{code}
> vertex gets cast to its own type. This cast can be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-178) TestPredicate lock has lots of boolean expressions to be simplified
[
https://issues.apache.org/jira/browse/GIRAPH-178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated GIRAPH-178:
-
Attachment: GIRAPH-178.patch
> TestPredicate lock has lots of boolean expressions to be simplified
> ---
>
> Key: GIRAPH-178
> URL: https://issues.apache.org/jira/browse/GIRAPH-178
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Trivial
> Labels: newbie
> Attachments: GIRAPH-178.patch
>
>
> TestPredicateLock.java has several instances of
> {code}assertTrue(gotPredicate == false);{code} (or {{== true}}) that can be
> simplified to more idiomatic Java.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-183) Add Claudio's FOSDEM presentation (slides and video) to the site
[ https://issues.apache.org/jira/browse/GIRAPH-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-183: Attachment: GIRAPH-183.diff Site update. > Add Claudio's FOSDEM presentation (slides and video) to the site > > > Key: GIRAPH-183 > URL: https://issues.apache.org/jira/browse/GIRAPH-183 > Project: Giraph > Issue Type: Improvement > Components: site >Reporter: Claudio Martella >Assignee: Claudio Martella >Priority: Trivial > Labels: newbie > Attachments: GIRAPH-183.diff > > > Presentation: > http://prezi.com/9ake_klzwrga/apache-giraph-distributed-graph-processing-in-the-cloud/ > Video: http://www.youtube.com/watch?v=3ZrqPEIPRe4, > http://www.youtube.com/watch?v=BmRaejKGeDM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: GIRAPH-153.patch > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > Attachments: GIRAPH-153.patch > > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-182) Provide SequenceFileVertexOutputFormat as an available OutputFormat
[ https://issues.apache.org/jira/browse/GIRAPH-182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Gollakota updated GIRAPH-182: - Attachment: GIRAPH-182-1.patch Implemented an abstract SequenceFileVertexOutputFormat. Provided an example implementation. > Provide SequenceFileVertexOutputFormat as an available OutputFormat > --- > > Key: GIRAPH-182 > URL: https://issues.apache.org/jira/browse/GIRAPH-182 > Project: Giraph > Issue Type: New Feature > Components: lib >Reporter: Pradeep Gollakota >Assignee: Pradeep Gollakota >Priority: Minor > Attachments: GIRAPH-182-1.patch > > > SequenceFile's are heavily used in Hadoop. We should provide > SequenceFileVertexOutputFormat. Since SequenceFileVertexInputFormat is > already provided, it makes sense to also provide a mirroring OutputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-181) Add Hadoop 1.0 profile to pom.xml
[ https://issues.apache.org/jira/browse/GIRAPH-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-181: - Attachment: GIRAPH-181.patch Add support for Hadoop 1.0.2 to README. Thanks for the reminder, Avery. Also added some whitespace formatting for consistency. > Add Hadoop 1.0 profile to pom.xml > - > > Key: GIRAPH-181 > URL: https://issues.apache.org/jira/browse/GIRAPH-181 > Project: Giraph > Issue Type: Improvement > Components: build >Affects Versions: 0.2.0 >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-181.patch, GIRAPH-181.patch > > > Hadoop 1.0.x is now considered the "current stable version" of Hadoop, > according to http://hadoop.apache.org/common/releases.html#Download . > This JIRA is to add support within Giraph's maven profile for the 1.0.x > Hadoop release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-181) Add Hadoop 1.0 profile to pom.xml
[ https://issues.apache.org/jira/browse/GIRAPH-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-181: - Attachment: GIRAPH-181.patch > Add Hadoop 1.0 profile to pom.xml > - > > Key: GIRAPH-181 > URL: https://issues.apache.org/jira/browse/GIRAPH-181 > Project: Giraph > Issue Type: Improvement > Components: build >Affects Versions: 0.2.0 >Reporter: Eugene Koontz >Assignee: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-181.patch > > > Hadoop 1.0.x is now considered the "current stable version" of Hadoop, > according to http://hadoop.apache.org/common/releases.html#Download . > This JIRA is to add support within Giraph's maven profile for the 1.0.x > Hadoop release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-177) SimplePageRankVertex has two redundant casts
[
https://issues.apache.org/jira/browse/GIRAPH-177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated GIRAPH-177:
-
Attachment: GIRAPH-177.patch
> SimplePageRankVertex has two redundant casts
>
>
> Key: GIRAPH-177
> URL: https://issues.apache.org/jira/browse/GIRAPH-177
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
>Priority: Trivial
> Labels: newbie
> Attachments: GIRAPH-177.patch
>
>
> {code}DoubleWritable maxPagerank =
> (DoubleWritable) maxAggreg.getAggregatedValue();
> LOG.info("aggregatedMaxPageRank=" + maxPagerank.get());
> DoubleWritable minPagerank =
> (DoubleWritable) minAggreg.getAggregatedValue();
> LOG.info("aggregatedMinPageRank=" + minPagerank.get());{code}
> Both MinAggregator and MaxAggregator are already parameterized on
> DoubleWritable, so it's not necessary to cast their functions' results.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-179) BspServiceMaster's PathFilter can be simplified
[
https://issues.apache.org/jira/browse/GIRAPH-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated GIRAPH-179:
-
Attachment: GIRAPH-179.patch
> BspServiceMaster's PathFilter can be simplified
> ---
>
> Key: GIRAPH-179
> URL: https://issues.apache.org/jira/browse/GIRAPH-179
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
>Priority: Trivial
> Labels: newbie
> Attachments: GIRAPH-179.patch
>
>
> {code} /**
>* Only get the finalized checkpoint files
>*/
> public static class FinalizedCheckpointPathFilter implements PathFilter {
> @Override
> public boolean accept(Path path) {
> if (path.getName().endsWith(
> BspService.CHECKPOINT_FINALIZED_POSTFIX)) {
> return true;
> }
> return false;
> }
> }{code}
> we can simplify this, eliminating the if statement and just returning the
> result of {{endsWith()}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-180) Publish SNAPSHOTs and released artifacts in the Maven repository
[ https://issues.apache.org/jira/browse/GIRAPH-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paolo Castagna updated GIRAPH-180: -- Description: Currently Giraph uses Maven to drive its build. However, no Maven artifacts nor SNAPSHOTs are published in the Apache Maven repository or Maven central. It would be useful to have Apache Giraph artifacts and SNAPSHOTs published and enable people to use Giraph without recompiling themselves. Right now users can checkout Giraph, mvn install it and use this for their dependency: org.apache.giraph giraph 0.2-SNAPSHOT So, it's not that bad, but it can be better. :-) was: Currently Giraph uses Maven to drive its build. However, no Maven artifacts nor SNAPSHOTs are published in the Apache Maven repository or Maven central. It would be useful to have Apache Giraph artifacts and SNAPSHOTs published and enable people to use Giraph without recompiling themselves. > Publish SNAPSHOTs and released artifacts in the Maven repository > > > Key: GIRAPH-180 > URL: https://issues.apache.org/jira/browse/GIRAPH-180 > Project: Giraph > Issue Type: Improvement > Components: build >Affects Versions: 0.1.0 >Reporter: Paolo Castagna >Priority: Minor > Original Estimate: 4h > Remaining Estimate: 4h > > Currently Giraph uses Maven to drive its build. > However, no Maven artifacts nor SNAPSHOTs are published in the Apache Maven > repository or Maven central. > It would be useful to have Apache Giraph artifacts and SNAPSHOTs published > and enable people to use Giraph without recompiling themselves. > Right now users can checkout Giraph, mvn install it and use this for their > dependency: > > org.apache.giraph > giraph > 0.2-SNAPSHOT > > So, it's not that bad, but it can be better. :-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
-removes unneeded from Facebook profile
-additional README content regarding maven profile usage
{code}
mvn -Phadoop_non_secure clean verify &&
mvn -Phadoop_facebook
-Dhadoop.jar.path=/Users/ekoontz/hadoop-20/build/hadoop-0.20.1-dev-core.jar
clean verify &&
mvn -Phadoop_0.20.203 clean verify &&
mvn clean verify &&
mvn -Phadoop_0.23 clean verify &&
mvn -Phadoop_trunk clean verify
{code}
succeeds.
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch,
> GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-85) Simplify return expression in RPCCommunications::getRPCProxy
[ https://issues.apache.org/jira/browse/GIRAPH-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-85: -- Attachment: GIRAPH-85-3.patch re-uploading this 85-3 patch while remembering to set Grant license button to 'on' !!! > Simplify return expression in RPCCommunications::getRPCProxy > > > Key: GIRAPH-85 > URL: https://issues.apache.org/jira/browse/GIRAPH-85 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: newbie > Fix For: 0.2.0 > > Attachments: GIRAPH-85-3.patch, GIRAPH-85-3.patch, GIRAPH-85.patch, > GIRAPH-85.patch > > > Twice in RPCCommunications::getRPCProxy a local variable, proxy, is created > and immediately returned. We can simplify this to just return the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
Avery, thanks a lot for your comments and trying out the patch. This latest
patch should work with the Facebook branch. I did:
{code}
git clone https://github.com/facebook/hadoop-20.git
{code}
and then {{ant package}}, and then I did in my giraph directory(with the latest
patch):
{code}
mvn -Phadoop_facebook
-Dhadoop.jar.path=/Users/ekoontz/hadoop-20/build/hadoop-0.20.1-dev-core.jar
clean verify
{code}
and this succeeded.
I noticed that although the Facebook branch does not have SASL or Security, it
does seem to have the inter-versioning support provided by HADOOP-6904. So, I
added another munge flag, called HADOOP_VERSIONED_RPC, per the following
updated matrix:
||profile||HADOOP_NON_SECURE||HADOOP_NON_SASL_RPC||HADOOP_NON_INTERVERSIONED_RPC||
|hadoop_non_secure|x|x|x|
|hadoop_facebook |x|x| |
|hadoop_0.20.203 | |x|x|
|hadoop_0.23 | | | |
|hadoop_trunk | | | |
I updated the README also in the patch.
This patch works for me for {{mvn -PX clean verify}} for all of the above
versions, and also with no -P given (which is equivalent to -Phadoop_0.20.203).
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch,
> GIRAPH-168.patch, GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-171) total time in MasterThread.run() is calculated incorrectly
[
https://issues.apache.org/jira/browse/GIRAPH-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-171:
-
Attachment: GIRAPH-171.patch
> total time in MasterThread.run() is calculated incorrectly
> --
>
> Key: GIRAPH-171
> URL: https://issues.apache.org/jira/browse/GIRAPH-171
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-171.patch
>
>
> While running PageMarkBenchMark, I was seeing in the output:
> {{graph.MasterThread(172): total: Took 1.3336739262910001E9 seconds.}}
> This was because currently, in {{MasterThread.run()}}, we have:
> {code}
> LOG.info("total: Took " +
> ((System.currentTimeMillis() / 1000.0d) -
> setupSecs) + " seconds.");
> {code}
> but it should be:
> {code}
>LOG.info("total: Took " +
>((System.currentTimeMillis() - startMillis) /
> 1000.0d) + " seconds.");
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
-Update hadoop_trunk to look for Hadoop version 3.0.0-SNAPSHOT
-change HADOOP_OLDRPC munge flag to more descriptive HADOOP_NON_SASL_RPC
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch,
> GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
Latest patch "flips" the set of munge directives from {HADOOP_NEWRPC,
HADOOP_SECURE} to {HADOOP_OLDRPC,HADOOP_NON_SECURE}. HADOOP_NON_SECURE is a
flag used currently in trunk, so this is a return back to the current trunk
state.
Making old-RPC-signature and non-secure be the exceptional cases seems to me
better because if we remove older Hadoop versions, we'll have also removed the
need for having any munge directives.
Please see the flag/profile matrix for this patch below:
||profile||HADOOP_OLDRPC||HADOOP_NON_SECURE||
|hadoop_non_secure|x|x|
|hadoop_0.20.203|x||
|hadoop_0.23| | |
|hadoop_trunk| | |
|hadoop_facebook| | |
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
As Avery mentioned, there are changes in the RPC function signatures that vary
across Hadoop versions.
Therefore, beside HADOOP_SECURE, this patch also adds a HADOOP_NEWRPC munge
flag. The flags are set in pom.xml as follows ('x' means the flag is set):
||profile||HADOOP_SECURE||HADOOP_NEWRPC||
|hadoop_non_secure|||
|hadoop_0.20.203|x||
|hadoop_0.23|x|x|
|hadoop_trunk|x|x|
|hadoop_facebook|x|x|
In my tests, "mvn -Phadoop_X clean verify" works for all X in
{0.20.203,0.23,trunk,non_secure}.
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch, GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> same conditional compilation requirements. It also makes it easier to add
> more maven profiles so that we can easily increase our hadoop version
> coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-85) Simplify return expression in RPCCommunications::getRPCProxy
[
https://issues.apache.org/jira/browse/GIRAPH-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-85:
--
Attachment: GIRAPH-85-3.patch
This adds the SupressWarnings("unchecked") annotation to several methods that
seem to need it for mvn verify to run successfully. It also simplifies one more
spot in RPCCommunications.java where variables are used to temporarily hold a
return value, but nothing is done with that value before returning it. This
brings the grand total to 3 places where this change was made.
I would like to throw the idea out there that assigning to the "proxy" and
other variables for a moment DOES have a clarity benefit that I would hate to
prune out of the codebase just to help me practice uploading patches, which I
have done on GIRAPH-87 and GIRAPH-157. If someone else wants to take a crack at
this or if you guys just want to leave it the way already is to forego this
extra practice, I will not be upset!
If not, I think this patch will work.
> Simplify return expression in RPCCommunications::getRPCProxy
>
>
> Key: GIRAPH-85
> URL: https://issues.apache.org/jira/browse/GIRAPH-85
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
> Labels: newbie
> Fix For: 0.2.0
>
> Attachments: GIRAPH-85-3.patch, GIRAPH-85.patch, GIRAPH-85.patch
>
>
> Twice in RPCCommunications::getRPCProxy a local variable, proxy, is created
> and immediately returned. We can simplify this to just return the value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-157) Vertex to perform graph coloring on simple, connected, undirected graphs and related test.
[ https://issues.apache.org/jira/browse/GIRAPH-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-157: --- Attachment: GIRAPH-157-2.patch This is an update to fix the initialization issue that IntIntNullIntVertex had (see GIRAPH-161) and that therefore my variation IntIntNullTextVertex carried with regard to possible null initialization of edges and messages. See GIRAPH-161 for details. I'm still looking for larger undirected, connected, simple graphs in line input format like [outboundEdge...] that we know the correct chromatic number of to test this thing on larger input graphs. So far, every graph I test it with is given a correct minimal coloring. Lets break this thing, anyone? > Vertex to perform graph coloring on simple, connected, undirected graphs and > related test. > -- > > Key: GIRAPH-157 > URL: https://issues.apache.org/jira/browse/GIRAPH-157 > Project: Giraph > Issue Type: Test > Components: examples, test >Affects Versions: 0.2.0 >Reporter: Eli Reisman >Assignee: Eli Reisman >Priority: Trivial > Labels: newbie > Attachments: GIRAPH-157-2.patch, GIRAPH-157.patch > > > Hi. I am attempting to learn the Hadoop and Giraph codebases and wanted to > write a simple client application for Giraph to help me learn the ins and > outs of it. This is a simple unit test and vertex modeled after the > ConnectedComponentsVertex and related test. The vertex test runs whenever you > run the "mvn test" or "mvn verify" suite of tests. When finished processing, > each vertex will have an integer value that is its color. > This is a pretty simple implementation, and although I have tested it on a > number of small graphs of varied trickiness and it seems to rapidly arrive at > a minimal coloring, its hard (for me at least) to guess which possible > coloring it will arrive at and I have no idea how it will do on really big > graphs yet without finding some more pre-colored larger test graphs to try it > on. Ideas anyone? > Anyway, it was fun to put this together, and I'd be happy to improve it or > receive some help or advice to further the cause. Thanks again, I am hoping > this will be the first of many (hopefully more useful) contributions! > Eli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-159) Case insensitive file/directory name matching will produce errors on M/R jar unpack.
[ https://issues.apache.org/jira/browse/GIRAPH-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-159: - Priority: Major (was: Minor) > Case insensitive file/directory name matching will produce errors on M/R jar > unpack. > - > > Key: GIRAPH-159 > URL: https://issues.apache.org/jira/browse/GIRAPH-159 > Project: Giraph > Issue Type: Bug > Components: build >Affects Versions: 0.2.0 > Environment: OSX 10.6.8 >Reporter: Brian Femiano > Attachments: GIRAPH-159.patch, compile.xml > > > This only seems to affect platforms where there can be a file/directory > naming conflicts > from case insensitive matches. > > I was able to reproduce running the pseudo-distributed unit tests within OSX. > This has affected other projects: > https://issues.apache.org/jira/browse/MAHOUT-780 > I've been able to reproduce this on my local OSX install with the following > error: > https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8 > Since LICENSE.txt contains the same content as the file LICENSE, I propose we > exclude any LICENSE matches found in the unpacked dependency jars > when the maven assembly phase hits 'jar-with-dependencies'. > I have a patch which moves the 'jar-with-dependencies' descriptor to an > external compile.xml file which has the proper excludes. This might also > come in handy down the road should any additional tweaks be needed to the > compile phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-167) mvn -Phadoop_non_secure clean verify fails
[
https://issues.apache.org/jira/browse/GIRAPH-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-167:
-
Attachment: GIRAPH-167.patch
> mvn -Phadoop_non_secure clean verify fails
> --
>
> Key: GIRAPH-167
> URL: https://issues.apache.org/jira/browse/GIRAPH-167
> Project: Giraph
> Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Labels: build, hadoop
> Attachments: GIRAPH-167.patch
>
>
> The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
> compile:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
> cannot find symbol
> symbol : variable versionID
> location: class org.apache.giraph.comm.RPCCommunications
> [INFO] 1 error
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: SetLongWritable.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: TableRootMarker.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: SetTextWritable.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: HBaseVertexOutputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: IdentifyAndMarkRoots.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: TableRootMarkerOutputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: TableRootMarkerInputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: DistributedCacheHelper.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: ComputeIsRoot.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: HBaseVertexInputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: AccumuloRootMarkerInputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: AccumuloVertexInputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: AccumuloVertexOutputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: AccumuloRootMarkerOutputFormat.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: (was: AccumuloRootMarker.java) > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Description:
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can eliminate
usage of munge in the future, but until then, we can mitigate the complexity by
consolidating the number of flags checked. This JIRA renames HADOOP_FACEBOOK to
HADOOP_SECURE, and removes usages of HADOOP, to handle the same conditional
compilation requirements. It also makes it easier to add more maven profiles so
that we can easily increase our hadoop version coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that we
can specify its dependencies separately from hadoop_trunk, because the hadoop
dependencies have changed between trunk and 0.205.0 - the former requires
hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of Facebook?
was:
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can eliminate
usage of munge in the future, but until then, we can mitigate the complexity by
consolidating the number of flags checked. This JIRA proposes a single flag,
HADOOP_SECURE, to handle the same conditional compilation requirements. It also
makes it easier to add more maven profiles so that we can easily increase our
hadoop version coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that we
can specify its dependencies separately from hadoop_trunk, because the hadoop
dependencies have changed between trunk and 0.205.0 - the former requires
hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of Facebook?
Summary: Simplify munge directive usage with new munge flag
HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP (was:
Simplify munge directive usage with new munge flag HADOOP_SECURE rather than
HADOOP_FACEBOOK)
> Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
> HADOOP_FACEBOOK) and remove usage of HADOOP
> -
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA renames
> HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
> sam
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Summary: Simplify munge directive usage with new munge flag HADOOP_SECURE
rather than HADOOP_FACEBOOK (was: Simplify munge directive usage with new
munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK and HADOOP_NON_SECURE)
> Simplify munge directive usage with new munge flag HADOOP_SECURE rather than
> HADOOP_FACEBOOK
>
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA proposes a
> single flag, HADOOP_SECURE, to handle the same conditional compilation
> requirements. It also makes it easier to add more maven profiles so that we
> can easily increase our hadoop version coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK and HADOOP_NON_SECURE
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
> Simplify munge directive usage with new munge flag HADOOP_SECURE rather than
> HADOOP_FACEBOOK and HADOOP_NON_SECURE
> --
>
> Key: GIRAPH-168
> URL: https://issues.apache.org/jira/browse/GIRAPH-168
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Attachments: GIRAPH-168.patch
>
>
> This JIRA relates to the mail thread here:
> http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
> Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and
> HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
> eliminate usage of munge in the future, but until then, we can mitigate the
> complexity by consolidating the number of flags checked. This JIRA proposes a
> single flag, HADOOP_SECURE, to handle the same conditional compilation
> requirements. It also makes it easier to add more maven profiles so that we
> can easily increase our hadoop version coverage.
> This patch modifies the existing hadoop_facebook profile to use the new
> HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
> It also adds a new hadoop maven profile, hadoop_trunk, which also sets
> HADOOP_SECURE.
> Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
> we can specify its dependencies separately from hadoop_trunk, because the
> hadoop dependencies have changed between trunk and 0.205.0 - the former
> requires hadoop-common, hadoop-mapreduce-client-core, and
> hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
> With this patch, the following passes:
> {code}
> mvn clean verify && mvn -Phadoop_trunk clean verify && mvn -Phadoop_0.20.203
> clean verify
> {code}
> Current problems:
> * I left in place the usage of HADOOP_NON_SECURE, but note that the profile
> that uses this is hadoop_non_secure, which fails to compile on trunk:
> https://issues.apache.org/jira/browse/GIRAPH-167 .
> * I couldn't get -Phadoop_facebook to work; does this work outside of
> Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-167) mvn -Phadoop_non_secure clean verify fails
[
https://issues.apache.org/jira/browse/GIRAPH-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-167:
-
Description:
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
cannot find symbol
symbol : variable versionID
location: class org.apache.giraph.comm.RPCCommunications
[INFO] 1 error
{code}
was:
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/RangePartitionOwner.java:[26,27]
package org.apache.hadoop.io does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/BasicPartitionOwner.java:[26,29]
package org.apache.hadoop.conf does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/BasicPartitionOwner.java:[27,29]
package org.apache.hadoop.conf does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/PartitionOwner.java:[22,27]
package org.apache.hadoop.io does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/PartitionOwner.java:[27,40]
cannot find symbol
symbol: class Writable
{code}
(more error messages follow)
> mvn -Phadoop_non_secure clean verify fails
> --
>
> Key: GIRAPH-167
> URL: https://issues.apache.org/jira/browse/GIRAPH-167
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>
> The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
> compile:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
> cannot find symbol
> symbol : variable versionID
> location: class org.apache.giraph.comm.RPCCommunications
> [INFO] 1 error
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-166) add '*.patch' to list of files that Apache Rat ignores
[ https://issues.apache.org/jira/browse/GIRAPH-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-166: - Attachment: (was: pom.xml) > add '*.patch' to list of files that Apache Rat ignores > -- > > Key: GIRAPH-166 > URL: https://issues.apache.org/jira/browse/GIRAPH-166 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Priority: Trivial > Attachments: GIRAPH-166.patch > > > Apache Rat will complain about "too many files without licenses" if it finds > any *.patch files in your working directory. Rat should ignore these since > they are temp files that aren't included in the distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-166) add '*.patch' to list of files that Apache Rat ignores
[ https://issues.apache.org/jira/browse/GIRAPH-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-166: - Attachment: GIRAPH-166.patch > add '*.patch' to list of files that Apache Rat ignores > -- > > Key: GIRAPH-166 > URL: https://issues.apache.org/jira/browse/GIRAPH-166 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Priority: Trivial > Attachments: GIRAPH-166.patch > > > Apache Rat will complain about "too many files without licenses" if it finds > any *.patch files in your working directory. Rat should ignore these since > they are temp files that aren't included in the distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-166) add '*.patch' to list of files that Apache Rat ignores
[ https://issues.apache.org/jira/browse/GIRAPH-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-166: - Attachment: pom.xml > add '*.patch' to list of files that Apache Rat ignores > -- > > Key: GIRAPH-166 > URL: https://issues.apache.org/jira/browse/GIRAPH-166 > Project: Giraph > Issue Type: Improvement >Reporter: Eugene Koontz >Priority: Trivial > Attachments: GIRAPH-166.patch > > > Apache Rat will complain about "too many files without licenses" if it finds > any *.patch files in your working directory. Rat should ignore these since > they are temp files that aren't included in the distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-165) checkstyle error: " 'conf' hides a field" on line 154 of GraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-165:
-
Summary: checkstyle error: " 'conf' hides a field" on line 154 of
GraphRunner (was: checkstyle error: 'conf'hides a field' on line 154 of
GraphRunner)
> checkstyle error: " 'conf' hides a field" on line 154 of GraphRunner
>
>
> Key: GIRAPH-165
> URL: https://issues.apache.org/jira/browse/GIRAPH-165
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>Priority: Minor
> Attachments: GIRAPH-165.patch
>
>
> full checkstyle error is
> {code}
> name="/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java">
> source="com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck"/>
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-165) checkstyle error: 'conf'hides a field' on line 154 of GraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-165:
-
Attachment: GIRAPH-165.patch
> checkstyle error: 'conf'hides a field' on line 154 of GraphRunner
> -
>
> Key: GIRAPH-165
> URL: https://issues.apache.org/jira/browse/GIRAPH-165
> Project: Giraph
> Issue Type: Bug
>Reporter: Eugene Koontz
>Priority: Minor
> Attachments: GIRAPH-165.patch
>
>
> full checkstyle error is
> {code}
> name="/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java">
> source="com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck"/>
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-164) fix 5 "Line is longer than 80 characters" style errors in GiraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-164:
-
Attachment: GIRAPH-164.patch
> fix 5 "Line is longer than 80 characters" style errors in GiraphRunner
> --
>
> Key: GIRAPH-164
> URL: https://issues.apache.org/jira/browse/GIRAPH-164
> Project: Giraph
> Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: Eugene Koontz
>Priority: Trivial
> Fix For: 0.2.0
>
> Attachments: GIRAPH-164.patch
>
>
> {code}
> name="/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java">
>source="com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck"/>
>source="com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck"/>
>source="com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck"/>
>source="com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck"/>
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-163) bin/giraph script overwrites CLASSPATH if "dev environment" detected (this also removes USER_JAR from CLASSPATH)
[ https://issues.apache.org/jira/browse/GIRAPH-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Heitmann updated GIRAPH-163: - Attachment: GIRAPH-163.patch this is a small patch to fix the described problem > bin/giraph script overwrites CLASSPATH if "dev environment" detected (this > also removes USER_JAR from CLASSPATH) > > > Key: GIRAPH-163 > URL: https://issues.apache.org/jira/browse/GIRAPH-163 > Project: Giraph > Issue Type: Improvement > Components: conf and scripts >Affects Versions: 0.1.0, 0.2.0 > Environment: current trunk of giraph, after running "mvn compile" (as > advised in the quick start guide). > Also Hadoop 1.0.1 was used. >Reporter: Benjamin Heitmann > Labels: newbie > Attachments: GIRAPH-163.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > If no ./lib dir is present, then the bin/giraph script assumes it is running > in a "dev environment". > This chooses an execution path through the bin/giraph script, which > overwrites the CLASSPATH variable instead of appending to it. > Incidentally, this also removes the name of the jar submitted by the user, > which got appended to CLASSPATH earlier in the script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-162) BspCase.setup() should catch FileNotFoundException thrown from org.apache.hadoop.fs.FileSystem.listStatus()
[ https://issues.apache.org/jira/browse/GIRAPH-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-162: - Description: In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to throw both FileNotFoundException and IOException. The former (FileNotFoundException) is currently not caught when BspCase.setup() looks for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete it. The listStatus() call throws FileNotException if this directory does not exist and causes several tests to fail when using Hadoop trunk. This exception should be caught and ignored during setup(), since it's not an error for this directory not to exist. (was: In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to throws both FileNotFoundException and IOException. The former (FileNotFoundException) is currently not caught when BspCase.setup() looks for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete it. The listStatus() call throws FileNotException if this directory does not exist and causes several tests to fail when using Hadoop trunk. This exception should be caught and ignored during setup(), since it's not an error for this directory not to exist.) > BspCase.setup() should catch FileNotFoundException thrown from > org.apache.hadoop.fs.FileSystem.listStatus() > --- > > Key: GIRAPH-162 > URL: https://issues.apache.org/jira/browse/GIRAPH-162 > Project: Giraph > Issue Type: Bug > Components: test >Affects Versions: 0.2.0 >Reporter: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-162.patch > > > In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to > throw both FileNotFoundException and IOException. The former > (FileNotFoundException) is currently not caught when BspCase.setup() looks > for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete > it. The listStatus() call throws FileNotException if this directory does not > exist and causes several tests to fail when using Hadoop trunk. This > exception should be caught and ignored during setup(), since it's not an > error for this directory not to exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-162) BspCase.setup() should catch FileNotFoundException thrown from org.apache.hadoop.fs.FileSystem.listStatus()
[ https://issues.apache.org/jira/browse/GIRAPH-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-162: - Attachment: GIRAPH-162.patch > BspCase.setup() should catch FileNotFoundException thrown from > org.apache.hadoop.fs.FileSystem.listStatus() > --- > > Key: GIRAPH-162 > URL: https://issues.apache.org/jira/browse/GIRAPH-162 > Project: Giraph > Issue Type: Bug > Components: test >Affects Versions: 0.2.0 >Reporter: Eugene Koontz > Fix For: 0.2.0 > > Attachments: GIRAPH-162.patch > > > In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to > throws both FileNotFoundException and IOException. The former > (FileNotFoundException) is currently not caught when BspCase.setup() looks > for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete > it. The listStatus() call throws FileNotException if this directory does not > exist and causes several tests to fail when using Hadoop trunk. This > exception should be caught and ignored during setup(), since it's not an > error for this directory not to exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-161) Handling null messages and edges when initializing IntIntNullIntVertex
[ https://issues.apache.org/jira/browse/GIRAPH-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dionysios Logothetis updated GIRAPH-161: Attachment: GIRAPH-161.patch > Handling null messages and edges when initializing IntIntNullIntVertex > -- > > Key: GIRAPH-161 > URL: https://issues.apache.org/jira/browse/GIRAPH-161 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.1.0 >Reporter: Dionysios Logothetis > Attachments: GIRAPH-161.patch > > > The initialize() method in org.apache.giraph.graph.IntIntNullIntVertex should > handle null messages or null edges. Especially initializing with null > messages is a common case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-160) Vertex reader that reads adjacency lists with no vertex and edge values associated
[ https://issues.apache.org/jira/browse/GIRAPH-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dionysios Logothetis updated GIRAPH-160: Attachment: GIRAPH-160.patch > Vertex reader that reads adjacency lists with no vertex and edge values > associated > -- > > Key: GIRAPH-160 > URL: https://issues.apache.org/jira/browse/GIRAPH-160 > Project: Giraph > Issue Type: New Feature > Components: lib >Affects Versions: 0.1.0 >Reporter: Dionysios Logothetis >Priority: Minor > Labels: features > Attachments: GIRAPH-160.patch > > > A very common format of graphs is adjacency lists with no values associated > to edges or vertices. For instance a line in the input can be of the type: > 1 2 3 > which represents a vertex with id 1 that has edges to vertices 2 and 3 with > no values associated. > I've created a vertex reader named AdjacencyListVertexReader which is > essentially a copy of the AdjacencyListVertexReader modified to handle this > format. It's an abstract class and subclasses can override the > defaultVertexValue() and defaultEdgeValue() methods to provide default values > for vertices and edges correspondingly (otherwise values are initialized to > null). > I've also created an example subclass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-159) Case insensitive file/directory name matching will produce errors on M/R jar unpack.
[ https://issues.apache.org/jira/browse/GIRAPH-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-159: - Attachment: compile.xml GIRAPH-159.patch > Case insensitive file/directory name matching will produce errors on M/R jar > unpack. > - > > Key: GIRAPH-159 > URL: https://issues.apache.org/jira/browse/GIRAPH-159 > Project: Giraph > Issue Type: Bug > Components: build >Affects Versions: 0.2.0 > Environment: OSX 10.6.8 >Reporter: Brian Femiano >Priority: Minor > Attachments: GIRAPH-159.patch, compile.xml > > > This only seems to affect platforms where there can be a file/directory > naming conflicts > from case insensitive matches. > > I was able to reproduce running the pseudo-distributed unit tests within OSX. > This has affected other projects: > https://issues.apache.org/jira/browse/MAHOUT-780 > I've been able to reproduce this on my local OSX install with the following > error: > https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8 > Since LICENSE.txt contains the same content as the file LICENSE, I propose we > exclude any LICENSE matches found in the unpacked dependency jars > when the maven assembly phase hits 'jar-with-dependencies'. > I have a patch which moves the 'jar-with-dependencies' descriptor to an > external compile.xml file which has the proper excludes. This might also > come in handy down the road should any additional tweaks be needed to the > compile phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-159) Case insensitive file/directory name matching will produce errors on M/R jar unpack.
[ https://issues.apache.org/jira/browse/GIRAPH-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-159: - Attachment: (was: GIRAPH-159.patch) > Case insensitive file/directory name matching will produce errors on M/R jar > unpack. > - > > Key: GIRAPH-159 > URL: https://issues.apache.org/jira/browse/GIRAPH-159 > Project: Giraph > Issue Type: Bug > Components: build >Affects Versions: 0.2.0 > Environment: OSX 10.6.8 >Reporter: Brian Femiano >Priority: Minor > > This only seems to affect platforms where there can be a file/directory > naming conflicts > from case insensitive matches. > > I was able to reproduce running the pseudo-distributed unit tests within OSX. > This has affected other projects: > https://issues.apache.org/jira/browse/MAHOUT-780 > I've been able to reproduce this on my local OSX install with the following > error: > https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8 > Since LICENSE.txt contains the same content as the file LICENSE, I propose we > exclude any LICENSE matches found in the unpacked dependency jars > when the maven assembly phase hits 'jar-with-dependencies'. > I have a patch which moves the 'jar-with-dependencies' descriptor to an > external compile.xml file which has the proper excludes. This might also > come in handy down the road should any additional tweaks be needed to the > compile phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-159) Case insensitive file/directory name matching will produce errors on M/R jar unpack.
[ https://issues.apache.org/jira/browse/GIRAPH-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-159: - Attachment: GIRAPH-159.patch > Case insensitive file/directory name matching will produce errors on M/R jar > unpack. > - > > Key: GIRAPH-159 > URL: https://issues.apache.org/jira/browse/GIRAPH-159 > Project: Giraph > Issue Type: Bug > Components: build >Affects Versions: 0.2.0 > Environment: OSX 10.6.8 >Reporter: Brian Femiano >Priority: Minor > Attachments: GIRAPH-159.patch > > > This only seems to affect platforms where there can be a file/directory > naming conflicts > from case insensitive matches. > > I was able to reproduce running the pseudo-distributed unit tests within OSX. > This has affected other projects: > https://issues.apache.org/jira/browse/MAHOUT-780 > I've been able to reproduce this on my local OSX install with the following > error: > https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8 > Since LICENSE.txt contains the same content as the file LICENSE, I propose we > exclude any LICENSE matches found in the unpacked dependency jars > when the maven assembly phase hits 'jar-with-dependencies'. > I have a patch which moves the 'jar-with-dependencies' descriptor to an > external compile.xml file which has the proper excludes. This might also > come in handy down the road should any additional tweaks be needed to the > compile phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-159) Case insensitive file/directory name matching will produce errors on M/R jar unpack.
[ https://issues.apache.org/jira/browse/GIRAPH-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-159: - Priority: Minor (was: Major) > Case insensitive file/directory name matching will produce errors on M/R jar > unpack. > - > > Key: GIRAPH-159 > URL: https://issues.apache.org/jira/browse/GIRAPH-159 > Project: Giraph > Issue Type: Bug > Components: build >Affects Versions: 0.2.0 > Environment: OSX 10.6.8 >Reporter: Brian Femiano >Priority: Minor > > This only seems to affect platforms where there can be a file/directory > naming conflicts > from case insensitive matches. > > I was able to reproduce running the pseudo-distributed unit tests within OSX. > This has affected other projects: > https://issues.apache.org/jira/browse/MAHOUT-780 > I've been able to reproduce this on my local OSX install with the following > error: > https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8 > Since LICENSE.txt contains the same content as the file LICENSE, I propose we > exclude any LICENSE matches found in the unpacked dependency jars > when the maven assembly phase hits 'jar-with-dependencies'. > I have a patch which moves the 'jar-with-dependencies' descriptor to an > external compile.xml file which has the proper excludes. This might also > come in handy down the road should any additional tweaks be needed to the > compile phase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-156) Users should be able to set simple 'custom arguments' via org.apache.giraph.GiraphRunner
[ https://issues.apache.org/jira/browse/GIRAPH-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated GIRAPH-156: -- Attachment: GIRAPH-156-2.patch > Users should be able to set simple 'custom arguments' via > org.apache.giraph.GiraphRunner > > > Key: GIRAPH-156 > URL: https://issues.apache.org/jira/browse/GIRAPH-156 > Project: Giraph > Issue Type: Improvement > Components: conf and scripts >Affects Versions: 0.1.0 >Reporter: Sebastian Schelter >Assignee: Sebastian Schelter > Fix For: 0.2.0 > > Attachments: GIRAPH-156-1.patch, GIRAPH-156-2.patch, GIRAPH-156.patch > > > Some vertices need custom arguments to run. The SimpleShortestPathsVertex for > example needs to know the source vertex for the computation which is saved in > the job's Configuration as _SimpleShortestPathsVertex.sourceId_. Users should > be able to apply such simple custom arguments via GiraphRunner. > I propose to add a new option _--customArguments_ where users can supply > arguments in the form _=,=_ for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-158) Support YARN (next generation MapReduce)
[ https://issues.apache.org/jira/browse/GIRAPH-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-158: - Attachment: GIRAPH-158.patch This patch passes "mvn verify" with pom.xml modified to use "0.24.0-SNAPSHOT" as hadoop.version. > Support YARN (next generation MapReduce) > > > Key: GIRAPH-158 > URL: https://issues.apache.org/jira/browse/GIRAPH-158 > Project: Giraph > Issue Type: New Feature >Reporter: Eugene Koontz > Attachments: GIRAPH-158.patch > > > YARN is a re-architecturing of the Hadoop MapReduce framework, described here: > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html > It would be good to offer support within Giraph for this framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-157) Vertex to perform graph coloring on simple, connected, undirected graphs and related test.
[ https://issues.apache.org/jira/browse/GIRAPH-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-157: --- Attachment: GIRAPH-157.patch > Vertex to perform graph coloring on simple, connected, undirected graphs and > related test. > -- > > Key: GIRAPH-157 > URL: https://issues.apache.org/jira/browse/GIRAPH-157 > Project: Giraph > Issue Type: Test > Components: examples, test >Affects Versions: 0.2.0 >Reporter: Eli Reisman >Assignee: Eli Reisman >Priority: Trivial > Labels: newbie > Attachments: GIRAPH-157.patch > > > Hi. I am attempting to learn the Hadoop and Giraph codebases and wanted to > write a simple client application for Giraph to help me learn the ins and > outs of it. This is a simple unit test and vertex modeled after the > ConnectedComponentsVertex and related test. The vertex test runs whenever you > run the "mvn test" or "mvn verify" suite of tests. When finished processing, > each vertex will have an integer value that is its color. > This is a pretty simple implementation, and although I have tested it on a > number of small graphs of varied trickiness and it seems to rapidly arrive at > a minimal coloring, its hard (for me at least) to guess which possible > coloring it will arrive at and I have no idea how it will do on really big > graphs yet without finding some more pre-colored larger test graphs to try it > on. Ideas anyone? > Anyway, it was fun to put this together, and I'd be happy to improve it or > receive some help or advice to further the cause. Thanks again, I am hoping > this will be the first of many (hopefully more useful) contributions! > Eli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-156) Users should be able to set simple 'custom arguments' via org.apache.giraph.GiraphRunner
[ https://issues.apache.org/jira/browse/GIRAPH-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated GIRAPH-156: -- Attachment: GIRAPH-156-1.patch had a missing whitespace in the last patch :) > Users should be able to set simple 'custom arguments' via > org.apache.giraph.GiraphRunner > > > Key: GIRAPH-156 > URL: https://issues.apache.org/jira/browse/GIRAPH-156 > Project: Giraph > Issue Type: Improvement > Components: conf and scripts >Affects Versions: 0.1.0 >Reporter: Sebastian Schelter >Assignee: Sebastian Schelter > Attachments: GIRAPH-156-1.patch, GIRAPH-156.patch > > > Some vertices need custom arguments to run. The SimpleShortestPathsVertex for > example needs to know the source vertex for the computation which is saved in > the job's Configuration as _SimpleShortestPathsVertex.sourceId_. Users should > be able to apply such simple custom arguments via GiraphRunner. > I propose to add a new option _--customArguments_ where users can supply > arguments in the form _=,=_ for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-156) Users should be able to set simple 'custom arguments' via org.apache.giraph.GiraphRunner
[ https://issues.apache.org/jira/browse/GIRAPH-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated GIRAPH-156: -- Attachment: GIRAPH-156.patch > Users should be able to set simple 'custom arguments' via > org.apache.giraph.GiraphRunner > > > Key: GIRAPH-156 > URL: https://issues.apache.org/jira/browse/GIRAPH-156 > Project: Giraph > Issue Type: Improvement > Components: conf and scripts >Affects Versions: 0.1.0 >Reporter: Sebastian Schelter >Assignee: Sebastian Schelter > Attachments: GIRAPH-156.patch > > > Some vertices need custom arguments to run. The SimpleShortestPathsVertex for > example needs to know the source vertex for the computation which is saved in > the job's Configuration as _SimpleShortestPathsVertex.sourceId_. Users should > be able to apply such simple custom arguments via GiraphRunner. > I propose to add a new option _--customArguments_ where users can supply > arguments in the form _=,=_ for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-154) Worker ports are not synched properly with its peers
[ https://issues.apache.org/jira/browse/GIRAPH-154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiwei Gu updated GIRAPH-154: - Attachment: GIRAPH-154.patch passed unit test and grid test. > Worker ports are not synched properly with its peers > > > Key: GIRAPH-154 > URL: https://issues.apache.org/jira/browse/GIRAPH-154 > Project: Giraph > Issue Type: Bug > Components: bsp >Affects Versions: 0.2.0 >Reporter: Zhiwei Gu >Assignee: Zhiwei Gu > Attachments: GIRAPH-154.patch > > > When worker trying multiple ports to setup the rpc server, the final port is > not synched with it's peer workers properly, and resulted in peer workers > send message to the default port. > Here is some logs: > > Base port: 34900 > > > log for worker 161: > > IPC Server handler 98 on 36061: starting > BasicRPCCommunications: Started RPC communication server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:36061 with 100 handlers and 199 > flush threads on bind attempt 1 > IPC Server handler 99 on 36061: starting > setup: Registering health of this worker... > getJobState: Job state already exists > (/_hadoopBsp/job_201203130609_14838/_masterJobState) > getApplicationAttempt: Node > /_hadoopBsp/job_201203130609_14838/_applicationAttemptsDir already exists! > getApplicationAttempt: Node > /_hadoopBsp/job_201203130609_14838/_applicationAttemptsDir already exists! > registerHealth: Created my health node for attempt=0, superstep=-1 with > /_hadoopBsp/job_201203130609_14838/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir/gsta32085.tan.ygrid.yahoo.com_161 > and workerInfo= Worker(hostname=gsta32085.tan.ygrid.yahoo.com, > MRpartition=161, port=35061) > process: partitionAssignmentsReadyChanged (partitions are assigned) > startSuperstep: Ready for computation on superstep -1 since worker selection > and vertex range assignments are done in > /_hadoopBsp/job_201203130609_14838/_applicationAttemptsDir/0/_superstepDir/-1/_partitionAssignments > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 0 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 1 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 2 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 3 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 4 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 5 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 6 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 7 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 8 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 9 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 10 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 11 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 12 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 13 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 14 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 15 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 16 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 17 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 18 time(s). > Retrying connect to server: > gsta32085.tan.ygrid.yahoo.com/10.216.148.47:35061. Already tried 19 time(s). > Retrying connect to server: > gsta32
[jira] [Updated] (GIRAPH-155) Allow creation of graph by adding edges that span multiple workers
[ https://issues.apache.org/jira/browse/GIRAPH-155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dionysios Logothetis updated GIRAPH-155: Description: Currently a graph is created only be adding vertices. The typical way is to read input text files line-by-line with each line describing a vertex (its value, its edges etc). The current API allows for the creation of a vertex only if all the information for the vertex is available in a single line. However, it's common to have graphs described in the form of edges. Edges might span multiple lines in an input file or even span multiple workers. The current API doesn't allow this. In the input superstep, a vertex must be created by a single worker. Instead, it should be possible for multiple workers to mutate the graph during the input superstep. This has the following implications: 1) Instead of just instantiating a vertex, a vertex reader should be able to do vertex addition and edge addition requests. 2) Multiple workers might try to create the same vertex. Any conflicts should be handled with a VertexResolver. So the resolver has to be instantiated before load time. was: Currently a graph is created only be adding vertices. The typical way is to read input text files line-by-line with each line describing a vertex (its value, its edges etc). The current API allows for the creation of a vertex only if all the information for the vertex is available in a single line. However, it's common to have graphs described in the form of edges. Edges might span multiple lines in an input file or even span multiple workers. The current API doesn't allow this. In the input superstep, a vertex must be created by a single worker. Instead, it should be possible for multiple workers to mutate the graph during the input superstep. This has the following implications: (i) Instead of just instantiating a vertex, a vertex reader should be able to do vertex addition and edge addition requests. (ii) Multiple workers might try to create the same vertex. Any conflicts should be handled with a VertexResolver. So the resolver has to be instantiated before load time. > Allow creation of graph by adding edges that span multiple workers > -- > > Key: GIRAPH-155 > URL: https://issues.apache.org/jira/browse/GIRAPH-155 > Project: Giraph > Issue Type: New Feature > Components: graph, lib >Affects Versions: 0.1.0 >Reporter: Dionysios Logothetis > > Currently a graph is created only be adding vertices. The typical way is to > read input text files line-by-line with each line describing a vertex (its > value, its edges etc). The current API allows for the creation of a vertex > only if all the information for the vertex is available in a single line. > However, it's common to have graphs described in the form of edges. Edges > might span multiple lines in an input file or even span multiple workers. The > current API doesn't allow this. In the input superstep, a vertex must be > created by a single worker. > Instead, it should be possible for multiple workers to mutate the graph > during the input superstep. > This has the following implications: > 1) Instead of just instantiating a vertex, a vertex reader should be able to > do vertex addition and edge addition requests. > 2) Multiple workers might try to create the same vertex. Any conflicts should > be handled with a VertexResolver. So the resolver has to be instantiated > before load time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Labels: (was: gir) Description: Four abstract classes that wrap their respective delegate input/output formats for easy hooks into vertex input format subclasses. I've included some sample programs that show two very simple graph algorithms. I have a graph generator that builds out a very simple directed structure, starting with a few 'root' nodes. Root nodes are defined as nodes which are not listed as a child anywhere in the graph. Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. Every vertex starts thinking it's a root. At superstep 0, send a message down to each child as a non-root notification. After superstep 1, only root nodes will have never been messaged. Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by bundling the notification logic followed by root node propagation. Once we've marked the appropriate nodes as roots, tell every child which roots it can be traced back to via one or more spanning trees. This will take N + 2 supersteps where N is the maximum number of hops from any root to any leaf, plus 2 supersteps for the initial root flagging. I've included all relevant code plus DistributedCacheHelper.java for recursive cache file and archive searches. It is more hadoop centric than giraph, but these jobs use it so I figured why not commit here. These have been tested through local JobRunner, pseudo-distributed on the aforementioned hardware, and full distributed on EC2. More details in the comments. was: Four abstract classes that wrap their respective delegate input/output formats for easy hooks into vertex input format subclasses. I've included some sample programs that show two very simple graph algorithms. I have a graph generator that builds out a very simple direct structure, starting with a few 'root' nodes. Root nodes are defined as nodes that is not listed as a child anywhere in the graph. Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. Every vertex starts thinking it's a root. At superstep 0, send a message down to each child as a non-root notification. After superstep 1, only root nodes will have never been messaged. Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by bundling the notification logic followed by root node propagation. Once we've marked the appropriate nodes as roots, tell every child which roots it can be traced back to via one or more spanning trees. This will take N + 2 supersteps where N is the maximum number of hops from any root to any leaf, plus 2 supersteps for the initial root flagging. I've included all relevant code plus DistributedCacheHelper.java for recursive cache file and archive searches. It is more hadoop centric than giraph in particular, but these jobs use it so I figured why not commit here. These have been tested through local JobRunner, pseudo-distributed on the aforementioned hardware, and full distributed on EC2. More details in the comments. > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > Attachments: AccumuloRootMarker.java, > AccumuloRootMarkerInputFormat.java, AccumuloRootMarkerOutputFormat.java, > AccumuloVertexInputFormat.java, AccumuloVertexOutputFormat.java, > ComputeIsRoot.java, DistributedCacheHelper.java, HBaseVertexInputFormat.java, > HBaseVertexOutputFormat.java, IdentifyAndMarkRoots.java, > SetLongWritable.java, SetTextWritable.java, TableRootMarker.java, > TableRootMarkerInputFormat.java, TableRootMarkerOutputFormat.java > > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple directed > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes which are not listed as a child anywhere in > the graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > hav
[jira] [Updated] (GIRAPH-153) HBase/Accumulo Input and Output formats
[ https://issues.apache.org/jira/browse/GIRAPH-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Femiano updated GIRAPH-153: - Attachment: TableRootMarkerOutputFormat.java TableRootMarkerInputFormat.java TableRootMarker.java SetTextWritable.java SetLongWritable.java IdentifyAndMarkRoots.java HBaseVertexOutputFormat.java HBaseVertexInputFormat.java DistributedCacheHelper.java ComputeIsRoot.java AccumuloVertexOutputFormat.java AccumuloVertexInputFormat.java AccumuloRootMarkerOutputFormat.java AccumuloRootMarkerInputFormat.java AccumuloRootMarker.java > HBase/Accumulo Input and Output formats > --- > > Key: GIRAPH-153 > URL: https://issues.apache.org/jira/browse/GIRAPH-153 > Project: Giraph > Issue Type: New Feature > Components: bsp >Affects Versions: 0.1.0 > Environment: Single host OSX 10.6.8 2.2Ghz Intel i7, 8GB >Reporter: Brian Femiano > Labels: gir > Attachments: AccumuloRootMarker.java, > AccumuloRootMarkerInputFormat.java, AccumuloRootMarkerOutputFormat.java, > AccumuloVertexInputFormat.java, AccumuloVertexOutputFormat.java, > ComputeIsRoot.java, DistributedCacheHelper.java, HBaseVertexInputFormat.java, > HBaseVertexOutputFormat.java, IdentifyAndMarkRoots.java, > SetLongWritable.java, SetTextWritable.java, TableRootMarker.java, > TableRootMarkerInputFormat.java, TableRootMarkerOutputFormat.java > > > Four abstract classes that wrap their respective delegate input/output > formats for > easy hooks into vertex input format subclasses. I've included some sample > programs that show two very simple graph > algorithms. I have a graph generator that builds out a very simple direct > structure, starting with a few 'root' nodes. > Root nodes are defined as nodes that is not listed as a child anywhere in the > graph. > Algorithm 1) AccumuloRootMarker.java --> Accumulo as read/write source. > Every vertex starts thinking it's a root. At superstep 0, send a message down > to each > child as a non-root notification. After superstep 1, only root nodes will > have never been messaged. > Algorithm 2) TableRootMarker --> HBase as read/write source. Expands on A1 by > bundling the notification logic followed by root node propagation. Once we've > marked the appropriate nodes as roots, tell every child which roots it can be > traced back to via one or more spanning trees. This will take N + 2 > supersteps where N is the maximum number of hops from any root to any leaf, > plus 2 supersteps for the initial root flagging. > I've included all relevant code plus DistributedCacheHelper.java for > recursive cache file and archive searches. It is more hadoop centric than > giraph in particular, but these jobs use it so I figured why not commit here. > These have been tested through local JobRunner, pseudo-distributed on the > aforementioned hardware, and full distributed on EC2. More details in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-85) Simplify return expression in RPCCommunications::getRPCProxy
[ https://issues.apache.org/jira/browse/GIRAPH-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-85: -- Attachment: GIRAPH-85.patch This is an improved version of GIRAPH-85.patch done in redone in git and meeting the "mvn test" and "mvn checkstyle:check" guidelines. Not tested on cluster setup. > Simplify return expression in RPCCommunications::getRPCProxy > > > Key: GIRAPH-85 > URL: https://issues.apache.org/jira/browse/GIRAPH-85 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Jakob Homan > Labels: newbie > Fix For: 0.2.0 > > Attachments: GIRAPH-85.patch, GIRAPH-85.patch > > > Twice in RPCCommunications::getRPCProxy a local variable, proxy, is created > and immediately returned. We can simplify this to just return the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-87) Simplify boolean expression in BspService::checkpointFrequencyMet
[
https://issues.apache.org/jira/browse/GIRAPH-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-87:
--
Attachment: GIRAPH-87.patch
This is an improved version of GIRAPH-87.patch that passes "mvn
checkstyle:check" and of course also "mvn test". Not tested on a cluster.
> Simplify boolean expression in BspService::checkpointFrequencyMet
> -
>
> Key: GIRAPH-87
> URL: https://issues.apache.org/jira/browse/GIRAPH-87
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
>Assignee: Eli Reisman
> Labels: newbie
> Attachments: GIRAPH-87.patch, GIRAPH-87.patch
>
>
> {noformat}if (superstep < firstCheckpoint) {
> return false;
> } else if (((superstep - firstCheckpoint) % checkpointFrequency) ==
> 0) {
> return true;
> } else {
> return false;
> }{noformat}
> can be simplified to just return the result of the else if evaluation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-85) Simplify return expression in RPCCommunications::getRPCProxy
[ https://issues.apache.org/jira/browse/GIRAPH-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Reisman updated GIRAPH-85: -- Attachment: GIRAPH-85.patch Simplifies 2 return statements without changing program logic or flow. Passes "mvn test" but not tested on cluster. > Simplify return expression in RPCCommunications::getRPCProxy > > > Key: GIRAPH-85 > URL: https://issues.apache.org/jira/browse/GIRAPH-85 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Jakob Homan > Labels: newbie > Fix For: 0.2.0 > > Attachments: GIRAPH-85.patch > > > Twice in RPCCommunications::getRPCProxy a local variable, proxy, is created > and immediately returned. We can simplify this to just return the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-87) Simplify boolean expression in BspService::checkpointFrequencyMet
[
https://issues.apache.org/jira/browse/GIRAPH-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-87:
--
Attachment: GIRAPH-87.patch
This is the patch for GIRAPH-87 JIRA newbie issue. Passed "mvn test", not
tested on cluster.
> Simplify boolean expression in BspService::checkpointFrequencyMet
> -
>
> Key: GIRAPH-87
> URL: https://issues.apache.org/jira/browse/GIRAPH-87
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
>Assignee: Eli Reisman
> Labels: newbie
> Attachments: GIRAPH-87.patch
>
>
> {noformat}if (superstep < firstCheckpoint) {
> return false;
> } else if (((superstep - firstCheckpoint) % checkpointFrequency) ==
> 0) {
> return true;
> } else {
> return false;
> }{noformat}
> can be simplified to just return the result of the else if evaluation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-150) PageRankBenchmark accesses wrong conf after GiraphJob is created
[ https://issues.apache.org/jira/browse/GIRAPH-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-150: --- Attachment: GIRAPH-150.patch Use the job.getConfiguration() instead of getConf() or else the vertex class doesn't get set properly. Also got rid of the other getConf() usage. Tested with 'mvn package' and 'mvn verify'. > PageRankBenchmark accesses wrong conf after GiraphJob is created > > > Key: GIRAPH-150 > URL: https://issues.apache.org/jira/browse/GIRAPH-150 > Project: Giraph > Issue Type: Bug >Reporter: Avery Ching >Assignee: Avery Ching > Attachments: GIRAPH-150.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-40) Adding checkstyle enforcement of Giraph code conventions
[
https://issues.apache.org/jira/browse/GIRAPH-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Avery Ching updated GIRAPH-40:
--
Attachment: GIRAPH-40.3.patch
Good suggestion, Jakob. I have addressed Jakob's comments by changing phases
from 'compile'->'verify' for checkstyle. This is the only change I made.
{noquote}
-compile
+verify
{noquote}
We need to require our users to have tested with 'mvn verify' (or 'mvn
install') before submitting diffs. This matches the rat approach.
> Adding checkstyle enforcement of Giraph code conventions
>
>
> Key: GIRAPH-40
> URL: https://issues.apache.org/jira/browse/GIRAPH-40
> Project: Giraph
> Issue Type: New Feature
>Reporter: Avery Ching
>Assignee: Avery Ching
>Priority: Minor
> Attachments: GIRAPH-40.2.patch, GIRAPH-40.3.patch, GIRAPH-40.patch,
> GIRAPH-40.patch
>
>
> Now that we have some code conventions (see GIRAPH-21), we should enforce
> them with a maven checkstyle plugin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-40) Adding checkstyle enforcement of Giraph code conventions
[ https://issues.apache.org/jira/browse/GIRAPH-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-40: -- Attachment: GIRAPH-40.2.patch I looked at the Hadoop checkstyle file and found it to be more orderly. I have taken this checkstyle.xml file and organized it better. Additionally, I added a few Hadoop checks that I found in the Hadoop checkstyle file that improve our code as well. Can someone please take a look? Thanks! > Adding checkstyle enforcement of Giraph code conventions > > > Key: GIRAPH-40 > URL: https://issues.apache.org/jira/browse/GIRAPH-40 > Project: Giraph > Issue Type: New Feature >Reporter: Avery Ching >Assignee: Avery Ching >Priority: Minor > Attachments: GIRAPH-40.2.patch, GIRAPH-40.patch, GIRAPH-40.patch > > > Now that we have some code conventions (see GIRAPH-21), we should enforce > them with a maven checkstyle plugin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-40) Adding checkstyle enforcement of Giraph code conventions
[ https://issues.apache.org/jira/browse/GIRAPH-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-40: -- Attachment: GIRAPH-40.patch As promised, here is the full patch. Due to its massive size, I am not posting this to reviewboard. Here are the details of what I did: * Created a checkstyle.xml file that follows our CODE_CONVENTIONS as best as possible. * Compiles will now fail if checkstyle guidelines are not met. * While checkstyle isn't comprehensive, it should reduce our reviewer overhead for formatting issues and common code style violations * Current source code (not test code) now meets checkstyle checks It passes both the local and MR unittests and also passes rat installation. Take a look at a few files. I don't recommend looking at everything since $ git diff HEAD^ | grep -P "^(\+|\-)" | wc -l 32848 Let's get this in soon to help us iterate faster and get rid of this technical debt! > Adding checkstyle enforcement of Giraph code conventions > > > Key: GIRAPH-40 > URL: https://issues.apache.org/jira/browse/GIRAPH-40 > Project: Giraph > Issue Type: New Feature >Reporter: Avery Ching >Assignee: Avery Ching >Priority: Minor > Attachments: GIRAPH-40.patch, GIRAPH-40.patch > > > Now that we have some code conventions (see GIRAPH-21), we should enforce > them with a maven checkstyle plugin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-148) giraph-site.xml needs Apache header
[ https://issues.apache.org/jira/browse/GIRAPH-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-148: --- Attachment: GIRAPH-148-b.patch Here's one copied and pasted from our pom.xml > giraph-site.xml needs Apache header > --- > > Key: GIRAPH-148 > URL: https://issues.apache.org/jira/browse/GIRAPH-148 > Project: Giraph > Issue Type: Bug > Components: conf and scripts >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-148-b.patch, GIRAPH-148.patch > > > I forgot to add the license to the conf file and now rat is failing... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-40) Adding checkstyle enforcement of Giraph code conventions
[ https://issues.apache.org/jira/browse/GIRAPH-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-40: -- Attachment: GIRAPH-40.patch I began with the commons-math checkstyle file and added and removed checks until I got to the closest interpretation of our CODE_CONVENTIONS. This is mostly pretty close to our CODE_CONVENTIONS with a few exceptions. I have formatted a few files and to show folks. If you guys like it, I'll format the entire Giraph codebase and make this checkstyle enforceable. I really believe that adding checkstyle will make our contribution process more smooth and allow us to work faster. > Adding checkstyle enforcement of Giraph code conventions > > > Key: GIRAPH-40 > URL: https://issues.apache.org/jira/browse/GIRAPH-40 > Project: Giraph > Issue Type: New Feature >Reporter: Avery Ching >Assignee: Avery Ching >Priority: Minor > Attachments: GIRAPH-40.patch > > > Now that we have some code conventions (see GIRAPH-21), we should enforce > them with a maven checkstyle plugin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-148) giraph-site.xml needs Apache header
[ https://issues.apache.org/jira/browse/GIRAPH-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-148: --- Summary: giraph-site.xml needs Apache header (was: giraph-site.xml needs Apache head) > giraph-site.xml needs Apache header > --- > > Key: GIRAPH-148 > URL: https://issues.apache.org/jira/browse/GIRAPH-148 > Project: Giraph > Issue Type: Bug > Components: conf and scripts >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-148.patch > > > I forgot to add the license to the conf file and now rat is failing... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-148) giraph-site.xml needs Apache header
[ https://issues.apache.org/jira/browse/GIRAPH-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-148: --- Attachment: GIRAPH-148.patch Quick patch... > giraph-site.xml needs Apache header > --- > > Key: GIRAPH-148 > URL: https://issues.apache.org/jira/browse/GIRAPH-148 > Project: Giraph > Issue Type: Bug > Components: conf and scripts >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-148.patch > > > I forgot to add the license to the conf file and now rat is failing... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-142) _hadoopBsp should be prefixable via configuration
[ https://issues.apache.org/jira/browse/GIRAPH-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-142: --- Attachment: GIRAPH-142-c.patch Updated with formatting changes. > _hadoopBsp should be prefixable via configuration > - > > Key: GIRAPH-142 > URL: https://issues.apache.org/jira/browse/GIRAPH-142 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-142-b.patch, GIRAPH-142-c.patch, GIRAPH-142.patch > > > In multitennant zookeeper clusters, it would be good to be able to specify > the base directory that's created for the _hadoopBsp znodes. This would also > fix the issue we have with creating that directory in the source root during > tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-142) _hadoopBsp should be prefixable via configuration
[ https://issues.apache.org/jira/browse/GIRAPH-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-142: --- Attachment: GIRAPH-142-b.patch Updated patch with check for starting slash. > _hadoopBsp should be prefixable via configuration > - > > Key: GIRAPH-142 > URL: https://issues.apache.org/jira/browse/GIRAPH-142 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-142-b.patch, GIRAPH-142.patch > > > In multitennant zookeeper clusters, it would be good to be able to specify > the base directory that's created for the _hadoopBsp znodes. This would also > fix the issue we have with creating that directory in the source root during > tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-142) _hadoopBsp should be prefixable via configuration
[ https://issues.apache.org/jira/browse/GIRAPH-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-142: --- Attachment: GIRAPH-142.patch Patch to add new config value, giraph.zkBaseZNode, that is the top-level for all giraph-created content on the ZK server. New unit test. Verified on running cluster as well. > _hadoopBsp should be prefixable via configuration > - > > Key: GIRAPH-142 > URL: https://issues.apache.org/jira/browse/GIRAPH-142 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-142.patch > > > In multitennant zookeeper clusters, it would be good to be able to specify > the base directory that's created for the _hadoopBsp znodes. This would also > fix the issue we have with creating that directory in the source root during > tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-145) Change partition request log level to debug rather than info
[
https://issues.apache.org/jira/browse/GIRAPH-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated GIRAPH-145:
---
Attachment: GIRAPH-145.patch
Quick patch to go down to debug level. Verified with tests and cluster run.
> Change partition request log level to debug rather than info
>
>
> Key: GIRAPH-145
> URL: https://issues.apache.org/jira/browse/GIRAPH-145
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
>Affects Versions: 0.2.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.2.0
>
> Attachments: GIRAPH-145.patch
>
>
> {code:title=BasicRPCCommunications.java|borderStyle=solid}
> if (LOG.isInfoEnabled()) {
> LOG.info("sendPartitionReq: Sending to " + rpcProxy.getName() +
> " " + addr + " from " + workerInfo +
> ", with partition " + partition);
> }{code}
> is too chatty. We're seeing thousands and sounds of these lines for larger
> graphs. This should be at debug level...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-144) GiraphJob should not extend Job (users should not be able to call Job methods like waitForCompletion or setMapper..etc)
[ https://issues.apache.org/jira/browse/GIRAPH-144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching updated GIRAPH-144: --- Attachment: GIRAPH-144.patch Here's the patch. > GiraphJob should not extend Job (users should not be able to call Job > methods like waitForCompletion or setMapper..etc) > > > Key: GIRAPH-144 > URL: https://issues.apache.org/jira/browse/GIRAPH-144 > Project: Giraph > Issue Type: Bug >Reporter: Dave >Assignee: Avery Ching > Attachments: GIRAPH-144.patch > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-139) Change PageRankBenchmark to be accessible via bin/giraph
[ https://issues.apache.org/jira/browse/GIRAPH-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-139: --- Attachment: GIRAPH-139-b.patch New patch that mains PageRankBenchmark as a separately runnable program, but one that can be quickly deleted once the example jar is sorted. > Change PageRankBenchmark to be accessible via bin/giraph > > > Key: GIRAPH-139 > URL: https://issues.apache.org/jira/browse/GIRAPH-139 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-139-b.patch, GIRAPH-139.patch > > > Currently the PageRankBenchmark has its own main and tool implementation and > is difficult to access from the bin/giraph script. It would be better if > everything were accessible via bin/giraph. The benchmark is particularly > problematic because it uses inner classes for its two actual Vertex > implementations, which have to be specified on the command line as their > .class name(ie > org.apache.giraph.benchmark.PageRankBenchmark$PageRankHashMapVertex) rather > than just with dots, as one would expect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-143) Add support for giraph to have a conf file
[ https://issues.apache.org/jira/browse/GIRAPH-143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-143: --- Component/s: conf and scripts > Add support for giraph to have a conf file > -- > > Key: GIRAPH-143 > URL: https://issues.apache.org/jira/browse/GIRAPH-143 > Project: Giraph > Issue Type: New Feature > Components: conf and scripts >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-143.patch > > > Currently one must provide all the Giraph-specific config values either via > the command line or snuck into another project's conf file. Any > self-respecting Hadoop ecosystem project should have its own conf file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-143) Add support for giraph to have a conf file
[ https://issues.apache.org/jira/browse/GIRAPH-143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-143: --- Attachment: GIRAPH-143.patch Patch adds a default, empty conf file and support in the code and scripts to use it. Verified by setting zookeeper configs and running jobs against it. > Add support for giraph to have a conf file > -- > > Key: GIRAPH-143 > URL: https://issues.apache.org/jira/browse/GIRAPH-143 > Project: Giraph > Issue Type: New Feature >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-143.patch > > > Currently one must provide all the Giraph-specific config values either via > the command line or snuck into another project's conf file. Any > self-respecting Hadoop ecosystem project should have its own conf file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-139) Change PageRankBenchmark to be accessible via bin/giraph
[ https://issues.apache.org/jira/browse/GIRAPH-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-139: --- Summary: Change PageRankBenchmark to be accessible via bin/giraph (was: Change PageRankBenchmark to use be accessible via bin/giraph) > Change PageRankBenchmark to be accessible via bin/giraph > > > Key: GIRAPH-139 > URL: https://issues.apache.org/jira/browse/GIRAPH-139 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.2.0 > > Attachments: GIRAPH-139.patch > > > Currently the PageRankBenchmark has its own main and tool implementation and > is difficult to access from the bin/giraph script. It would be better if > everything were accessible via bin/giraph. The benchmark is particularly > problematic because it uses inner classes for its two actual Vertex > implementations, which have to be specified on the command line as their > .class name(ie > org.apache.giraph.benchmark.PageRankBenchmark$PageRankHashMapVertex) rather > than just with dots, as one would expect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-139) Change PageRankBenchmark to use be accessible via bin/giraph
[
https://issues.apache.org/jira/browse/GIRAPH-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated GIRAPH-139:
---
Attachment: GIRAPH-139.patch
Patch that makes the page rank benchmark accessible from bin/giraph. The
default is the edgelist-based vertex, but the hashmap-based vertex is available
as a separate class. Calling the example is a bit hairy, since it's now not
restricted to just the pseudoinputformat:
{noformat} bin/giraph \n
-DPageRankBenchmark.superstepCount=200 \n
-DpseudoRandomVertexReader.aggregateVertices=220 \n
-DpseudoRandomVertexReader.edgesPerVertex=37 \n
lib/giraph-0.2-SNAPSHOT.jar \n
org.apache.giraph.benchmark.HashMapPageRankBenchmark \n -w 10 \n
-if org.apache.giraph.benchmark.PseudoRandomVertexInputFormat \n
-of org.apache.giraph.lib.AdjacencyListTextVertexOutputFormat \n
-op benchmark_results
{noformat}
I'm thinking that allowing vertices to provide default in/outputformat via
annotations may be a way to avoid some of this extra text.
Tests, javadoc and rat passes (except rat complains about CODE_CONVENTIONS,
which is out of scope of this JIRA).
Once this is committed, we need to update the wiki...
> Change PageRankBenchmark to use be accessible via bin/giraph
>
>
> Key: GIRAPH-139
> URL: https://issues.apache.org/jira/browse/GIRAPH-139
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-139.patch
>
>
> Currently the PageRankBenchmark has its own main and tool implementation and
> is difficult to access from the bin/giraph script. It would be better if
> everything were accessible via bin/giraph. The benchmark is particularly
> problematic because it uses inner classes for its two actual Vertex
> implementations, which have to be specified on the command line as their
> .class name(ie
> org.apache.giraph.benchmark.PageRankBenchmark$PageRankHashMapVertex) rather
> than just with dots, as one would expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-141) mulitgraph support in giraph
[
https://issues.apache.org/jira/browse/GIRAPH-141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
André Kelpe updated GIRAPH-141:
---
Description:
The current vertex API only supports simple graphs, meaning that there can only
ever be one edge between two vertices. Many graphs like the road network are in
fact multigraphs, where many edges can connect two vertices at the same time.
Support for this could be added by introducing an Iterator
getEdgeValue() or a similar construct. Maybe introducing a slim object like a
Connector between the edge and the vertex is also a good idea, so that you
could do something like:
{code}
for (final Connector conn: getEdgeValues(){
final EdgeWritable edge = conn.getEdge();
final VertexWritable otherVertex = conn.getOther();
doInterestingStuff(otherVertex);
doMoreInterestingStuff(edge);
}
{code}
was:
The current vertex API only supports simple graphs, meaning that there can only
ever be one edge between two vertices. Many graphs like the road network are in
fact multigraphs, where many edges can connect two vertices at the same time.
Support for this could be added by introducing an Iterator
getEdgeValue() or a similar construct. Maybe introducing a slim object like a
Connector between the edge and the vertex is also a good idea, so that you
could do something like:
{code}
for (final Connector conn: getEdgeValues(){
final EdgeWritable edge = conn.getEdge();
final VertexWritable otherVertex = conn.getOther();
// do interesting stuff
}
{code}
> mulitgraph support in giraph
>
>
> Key: GIRAPH-141
> URL: https://issues.apache.org/jira/browse/GIRAPH-141
> Project: Giraph
> Issue Type: Improvement
> Components: graph
>Reporter: André Kelpe
>
> The current vertex API only supports simple graphs, meaning that there can
> only ever be one edge between two vertices. Many graphs like the road network
> are in fact multigraphs, where many edges can connect two vertices at the
> same time.
> Support for this could be added by introducing an Iterator
> getEdgeValue() or a similar construct. Maybe introducing a slim object like a
> Connector between the edge and the vertex is also a good idea, so that you
> could do something like:
> {code}
> for (final Connector conn: getEdgeValues(){
> final EdgeWritable edge = conn.getEdge();
> final VertexWritable otherVertex = conn.getOther();
> doInterestingStuff(otherVertex);
> doMoreInterestingStuff(edge);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-141) mulitgraph support in giraph
[
https://issues.apache.org/jira/browse/GIRAPH-141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
André Kelpe updated GIRAPH-141:
---
Description:
The current vertex API only supports simple graphs, meaning that there can only
ever be one edge between two vertices. Many graphs like the road network are in
fact multigraphs, where many edges can connect two vertices at the same time.
Support for this could be added by introducing an Iterator
getEdgeValue() or a similar construct. Maybe introducing a slim object like a
Connector between the edge and the vertex is also a good idea, so that you
could do something like:
{code}
for (final Connector conn: getEdgeValues(){
final EdgeWritable edge = conn.getEdge();
final VertexWritable otherVertex = conn.getOther();
// do interesting stuff
}
{code}
was:
The current vertex API only supports simple graphs, meaning that there can only
ever be one edge between two vertices. Many graphs like the road network are in
fact multigraphs, where many edges can connect two vertices at the same time.
Support for this could be added by introducing an Iterator
getEdgeValue() or a similar construct. Maybe introducing a slim object like a
Connector between the edge and the vertex is also a good idea, so that you
could do something like:
for (final Connector conn: getEdgeValues(){
final EdgeWritable edge = conn.getEdge();
final VertexWritable otherVertex = conn.getOther();
// do interesting stuff
}
> mulitgraph support in giraph
>
>
> Key: GIRAPH-141
> URL: https://issues.apache.org/jira/browse/GIRAPH-141
> Project: Giraph
> Issue Type: Improvement
> Components: graph
>Reporter: André Kelpe
>
> The current vertex API only supports simple graphs, meaning that there can
> only ever be one edge between two vertices. Many graphs like the road network
> are in fact multigraphs, where many edges can connect two vertices at the
> same time.
> Support for this could be added by introducing an Iterator
> getEdgeValue() or a similar construct. Maybe introducing a slim object like a
> Connector between the edge and the vertex is also a good idea, so that you
> could do something like:
> {code}
> for (final Connector conn: getEdgeValues(){
> final EdgeWritable edge = conn.getEdge();
> final VertexWritable otherVertex = conn.getOther();
> // do interesting stuff
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-136) Error message for bin/giraph could be improved
[
https://issues.apache.org/jira/browse/GIRAPH-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated GIRAPH-136:
---
Summary: Error message for bin/giraph could be improved (was: Erorr
message for bin/giraph could be improved)
> Error message for bin/giraph could be improved
> --
>
> Key: GIRAPH-136
> URL: https://issues.apache.org/jira/browse/GIRAPH-136
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.1.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.2.0
>
> Attachments: GIRAPH-136-b.patch, GIRAPH-136.patch
>
>
> Currently when one just runs bin/giraph without the required jar, the message
> isn't very helpful:
> {noformat}[tardis giraph-0.1]$ bin/giraph
> Can't find user jar to execute.{noformat}
> It would be better to have a more in-depth message explaining Giraph and what
> is expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-130) Fix Javadoc warnings
[
https://issues.apache.org/jira/browse/GIRAPH-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated GIRAPH-130:
---
Attachment: GIRAPH-130.patch
Fixes all trunk's present javadocs today.
@Avery - Perhaps once you have a QA Buildbot like other projects have, you can
use the javadoc hooks in it to +1/-1 a patch?
> Fix Javadoc warnings
>
>
> Key: GIRAPH-130
> URL: https://issues.apache.org/jira/browse/GIRAPH-130
> Project: Giraph
> Issue Type: Bug
>Reporter: Jakob Homan
>Priority: Minor
> Labels: newbie
> Attachments: GIRAPH-130.patch
>
>
> We've accumulated a fair number of javadoc warnings recently:
> {noformat}[WARNING] Javadoc Warnings
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:129:
> warning - @param argument "superstep" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java:84:
> warning - @param argument "vertexIndex" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java:84:
> warning - @param argument "msgList" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java:32:
> warning - Tag @link: reference not found: VertexIdMessage
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java:46:
> warning - Tag @link: reference not found: messages
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java:46:
> warning - Tag @link: reference not found: messages
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/AggregatorWriter.java:60:
> warning - @param argument "map" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/GiraphJob.java:432:
> warning - @param argument "graphPartitionerClass" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java:46:
> warning - Tag @link: reference not found: messages
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/partition/MasterGraphPartitioner.java:62:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/partition/MasterGraphPartitioner.java:62:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/partition/MasterGraphPartitioner.java:62:
> warning - @param argument "availableWorkerInfos" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/graph/partition/PartitionBalancer.java:176:
> warning - @param argument "allPartitionStatsList" is not a parameter name.
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java:32:
> warning - Tag @link: reference not found: VertexIdMessage
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java:32:
> warning - Tag @link: reference not found: VertexIdMessage
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: reference not found: GraphPartitioner
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java:32:
> warning - Tag @link: reference not found: VertexIdMessage
> [WARNING]
> /Users/jhoman/repos/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java:146:
> warning - Tag @link: referenc
[jira] [Updated] (GIRAPH-137) De-duplicate pagerank implementation in PageRankBenchmark
[
https://issues.apache.org/jira/browse/GIRAPH-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated GIRAPH-137:
---
Attachment: GIRAPH-137.patch
Subclassing having proven tricky to do (seems like a multiple inheritance
situation?) I've tried to reuse via a static function. Is this OK or plain
silly?
> De-duplicate pagerank implementation in PageRankBenchmark
> -
>
> Key: GIRAPH-137
> URL: https://issues.apache.org/jira/browse/GIRAPH-137
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Minor
> Labels: newbie
> Attachments: GIRAPH-137.patch
>
>
> Currently in PageRankBenchmark we have the code for pagerank duplicated in
> each of the implementations of Vertex:
> {noformat}public static class PageRankHashMapVertex extends HashMapVertex<
> LongWritable, DoubleWritable, DoubleWritable, DoubleWritable> {
> @Override
> public void compute(Iterator msgIterator) {
> if (getSuperstep() >= 1) {
> double sum = 0;
> while (msgIterator.hasNext()) {
> sum += msgIterator.next().get();
> }
> DoubleWritable vertexValue =
> new DoubleWritable((0.15f / getNumVertices()) + 0.85f *
>sum);
> setVertexValue(vertexValue);
> }
> if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT, -1)) {
> long edges = getNumOutEdges();
> sendMsgToAllEdges(
> new DoubleWritable(getVertexValue().get() / edges));
> } else {
> voteToHalt();
> }
> }
> }
> public static class PageRankEdgeListVertex extends EdgeListVertex<
> LongWritable, DoubleWritable, DoubleWritable, DoubleWritable> {
> @Override
> public void compute(Iterator msgIterator) {
> if (getSuperstep() >= 1) {
> double sum = 0;
> while (msgIterator.hasNext()) {
> sum += msgIterator.next().get();
> }
> DoubleWritable vertexValue =
> new DoubleWritable((0.15f / getNumVertices()) + 0.85f *
>sum);
> setVertexValue(vertexValue);
> }
> if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT, -1)) {
> long edges = getNumOutEdges();
> sendMsgToAllEdges(
> new DoubleWritable(getVertexValue().get() / edges));
> } else {
> voteToHalt();
> }
> }
> }{noformat}
> This code can be consolidated into private class and the two implementations
> just extend that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-133) Typo in JavaDoc in BspCase::remove
[
https://issues.apache.org/jira/browse/GIRAPH-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated GIRAPH-133:
---
Attachment: GIRAPH-133.patch
> Typo in JavaDoc in BspCase::remove
> --
>
> Key: GIRAPH-133
> URL: https://issues.apache.org/jira/browse/GIRAPH-133
> Project: Giraph
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Trivial
> Labels: newbie
> Attachments: GIRAPH-133.patch
>
>
> Configuration is spelled wrong in the javadoc:
> {noformat}/**
> * Helper method to remove a path if it exists.
> *
> * @param conf Configutation
> * @param path Path to remove
> * @throws IOException
> */
> public static void remove(Configuration conf, Path path)
> throws IOException {
> FileSystem hdfs = FileSystem.get(conf);
> hdfs.delete(path, true);
> }{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-131) enable creation of test-jars to simplify testing in downstream projects
[
https://issues.apache.org/jira/browse/GIRAPH-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Charles updated GIRAPH-131:
Attachment: GIRAPH-131-source-test-jar.patch
GIRAPH-131-source-test-jar.patch allows the deployment of the test-jar sources
(see also GIRAPH-129).
No much, but can be useful to run tests from a 3rd party project and debug in
the giraph sources.
> enable creation of test-jars to simplify testing in downstream projects
> ---
>
> Key: GIRAPH-131
> URL: https://issues.apache.org/jira/browse/GIRAPH-131
> Project: Giraph
> Issue Type: Improvement
>Reporter: André Kelpe
>Assignee: André Kelpe
>Priority: Minor
> Fix For: 0.1.0
>
> Attachments: GIRAPH-131-source-test-jar.patch, GIRAPH-131.patch
>
>
> Attached patch enables the creation of test-jars, which are the tests
> packaged in a separate jar file. This makes it possible to use the
> super-useful test infrastructure in MockUtils in downstream projects. If you
> add the patch, you will get a ${giraph.version}-tests.jar, which can be used
> for downstream testing like this:
>
> org.apache.giraph
> giraph
> ${giraph.version}
> test-jar
> test
>
> P.S.: The patch also resets the version to 0.1-SNAPSHOT as discussed in
> GIRAPH-129
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-136) Erorr message for bin/giraph could be improved
[
https://issues.apache.org/jira/browse/GIRAPH-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated GIRAPH-136:
---
Attachment: GIRAPH-136-b.patch
Here's a version that tries to be a bit smarter. If there's no lib directory,
it checks for a target directory (if target doesn't exist, it exits) and loads
the giraph jar from there and sets the classpath via maven (as described above).
This will work for dev enviroments with a hadoop instance. Invariably, this
won't work for someone and need to be modified more, but that's how these
scripts end up becoming so convoluted.
> Erorr message for bin/giraph could be improved
> --
>
> Key: GIRAPH-136
> URL: https://issues.apache.org/jira/browse/GIRAPH-136
> Project: Giraph
> Issue Type: Improvement
>Affects Versions: 0.1.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.2.0
>
> Attachments: GIRAPH-136-b.patch, GIRAPH-136.patch
>
>
> Currently when one just runs bin/giraph without the required jar, the message
> isn't very helpful:
> {noformat}[tardis giraph-0.1]$ bin/giraph
> Can't find user jar to execute.{noformat}
> It would be better to have a more in-depth message explaining Giraph and what
> is expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-120) Add Sebastian Schelter to site
[ https://issues.apache.org/jira/browse/GIRAPH-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated GIRAPH-120: -- Attachment: GIRAPH-120.patch > Add Sebastian Schelter to site > -- > > Key: GIRAPH-120 > URL: https://issues.apache.org/jira/browse/GIRAPH-120 > Project: Giraph > Issue Type: Task >Affects Versions: 0.1.0 >Reporter: Sebastian Schelter >Assignee: Sebastian Schelter > Fix For: 0.1.0 > > Attachments: GIRAPH-120.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-117) DefaultWorkerContext should preserve the method signatures of WorkerContext
[ https://issues.apache.org/jira/browse/GIRAPH-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-117: --- Fix Version/s: 0.2.0 @Sebastian - this is ready to go. Want to commit? > DefaultWorkerContext should preserve the method signatures of WorkerContext > --- > > Key: GIRAPH-117 > URL: https://issues.apache.org/jira/browse/GIRAPH-117 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Sebastian Schelter >Assignee: Sebastian Schelter >Priority: Trivial > Fix For: 0.2.0 > > Attachments: GIRAPH-117.patch > > > DefaultWorkerContext.preApplication() swallows the InstantiationException and > IllegalAccessException of WorkerContext.preApplication(). These should be > preserved for applications that want to register an aggregator in this method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
