[jira] [Commented] (GIRAPH-72) Running multiple Giraph jobs on the same cluster can lead to port collisions

2011-11-02 Thread Avery Ching (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142870#comment-13142870
 ] 

Avery Ching commented on GIRAPH-72:
---

Ah, you can set the base rpc port by the way, sorry forgot to mention.

/** Initial port to start using for the RPC communication */
public static final String RPC_INITIAL_PORT = "giraph.rpcInitialPort";
/** Default port to start using for the RPC communication */
public static final int RPC_INITIAL_PORT_DEFAULT = 3;

The right thing is to fix the failures early on in the application in general 
(something that doesn't kick in until after the first checkpoint currently).  
Let's address that issue after GIRAPH-11.  It's a monstrous change that is 
working right now (passed local and MR unittests) and will change a lot on how 
to do that exactly.  Thanks for filing the issue, agreed it's a problem.

> Running multiple Giraph jobs on the same cluster can lead to port collisions
> 
>
> Key: GIRAPH-72
> URL: https://issues.apache.org/jira/browse/GIRAPH-72
> Project: Giraph
>  Issue Type: Bug
>  Components: lib, zookeeper
>Affects Versions: 0.70.0
> Environment: production hadoop cluster, in-process ZK.
>Reporter: Jake Mannix
>
> Had a Giraph mini-hackathon at work today, and lots of us launched 
> simultaneous test jobs at the same time, and often ran into the following 
> collision:
> --
> startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1
> 2-Nov-2011 23:40:08
> java.net.BindException: Problem binding to /:3 : 
> Address already in use
>   at org.apache.hadoop.ipc.Server.bind(Server.java:196)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:259)
>   at org.apache.hadoop.ipc.Server.(Server.java:1039)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:492)
>   at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
>   at 
> org.apache.giraph.comm.RPCCommunications.getRPCServer(RPCCommunications.java:99)
>   at 
> org.apache.giraph.comm.BasicRPCCommunications.(BasicRPCCommunications.java:362)
>   at org.apache.giraph.comm.RPCCommunications.(RPCCommunications.java:71)
>   at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:570)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>   at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:194)
>   ... 12 more
> 
> The job then simply hung.  What it should do, I'd imagine, is at a bare 
> minimum, catch this exception and allow the task to die quickly so it can get 
> retried on another machine, or better yet, allow for a command-line arg at 
> startup (and then passed into the Configuration) decide what ports to use.  
> Best yet, something automagic which allows multiple GraphMappers on the same 
> machine without manually picking ports (pick one at random and store it in 
> zookeeper?  but then what about the in-process zookeeper...) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-72) Running multiple Giraph jobs on the same cluster can lead to port collisions

2011-11-02 Thread Jake Mannix (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142862#comment-13142862
 ] 

Jake Mannix commented on GIRAPH-72:
---

Well, as I said, at a bare minimum, if you're on a cluster where you can't just 
grab all the slots on a box, being able to manually set the ports used would 
mean that you only collide with *yourself* in a given task, because everybody 
is using different ports on different jobs (or maybe even by default generate a 
random port based on the job id if not specified?).  More clever things would 
be great, of course, but I'm not sure what the best ones would be.  Even less 
clever things would be helpful: on this specific exception, die early in a way 
that sends your task to another box.  Is that possible/easy?

> Running multiple Giraph jobs on the same cluster can lead to port collisions
> 
>
> Key: GIRAPH-72
> URL: https://issues.apache.org/jira/browse/GIRAPH-72
> Project: Giraph
>  Issue Type: Bug
>  Components: lib, zookeeper
>Affects Versions: 0.70.0
> Environment: production hadoop cluster, in-process ZK.
>Reporter: Jake Mannix
>
> Had a Giraph mini-hackathon at work today, and lots of us launched 
> simultaneous test jobs at the same time, and often ran into the following 
> collision:
> --
> startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1
> 2-Nov-2011 23:40:08
> java.net.BindException: Problem binding to /:3 : 
> Address already in use
>   at org.apache.hadoop.ipc.Server.bind(Server.java:196)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:259)
>   at org.apache.hadoop.ipc.Server.(Server.java:1039)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:492)
>   at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
>   at 
> org.apache.giraph.comm.RPCCommunications.getRPCServer(RPCCommunications.java:99)
>   at 
> org.apache.giraph.comm.BasicRPCCommunications.(BasicRPCCommunications.java:362)
>   at org.apache.giraph.comm.RPCCommunications.(RPCCommunications.java:71)
>   at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:570)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>   at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:194)
>   ... 12 more
> 
> The job then simply hung.  What it should do, I'd imagine, is at a bare 
> minimum, catch this exception and allow the task to die quickly so it can get 
> retried on another machine, or better yet, allow for a command-line arg at 
> startup (and then passed into the Configuration) decide what ports to use.  
> Best yet, something automagic which allows multiple GraphMappers on the same 
> machine without manually picking ports (pick one at random and store it in 
> zookeeper?  but then what about the in-process zookeeper...) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: GIRAPH-64

2011-11-02 Thread Jake Mannix 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2700/#review3023
---


It would be super cool if, like in Mahout and Hadoop, we could print out the 
set of Vertex impls / algorithms / actions available.  In Mahout, we have a 
properties file which we keep updating.  Even nicer (maybe?) would be to just 
search the classpath for all classes which implement BasicVertex, and list 
them.  Even cooler: have a new abstract method on BasicVertex "String 
getHelp()" which GiraphRunner could run via reflection to get the help output 
for the task.


/trunk/src/main/java/org/apache/giraph/GiraphRunner.java


Do these both need to be set?  -of is certainly not required by the 
framework.  



/trunk/src/main/java/org/apache/giraph/GiraphRunner.java


both are required?  below you only use them if they are supplied.



/trunk/src/main/java/org/apache/giraph/GiraphRunner.java


ip vs i ?



/trunk/src/main/java/org/apache/giraph/GiraphRunner.java


!isQuiet vs verbose... we're defaulting to "verbose" in this case?


- Jake


On 2011-11-03 03:25:11, Jakob Homan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2700/
> ---
> 
> (Updated 2011-11-03 03:25:11)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> create giraphrunner
> 
> 
> Diffs
> -
> 
>   /trunk/bin/giraph PRE-CREATION 
>   /trunk/pom.xml 1196926 
>   /trunk/src/main/assembly/assembly.xml PRE-CREATION 
>   /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION 
>   /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926 
> 
> Diff: https://reviews.apache.org/r/2700/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jakob
> 
>

Re: Review Request: GIRAPH-64

2011-11-02 Thread Jakob Homan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2700/
---

(Updated 2011-11-03 03:25:11.909806)


Review request for giraph.


Summary (updated)
---

create giraphrunner


Diffs
-

  /trunk/bin/giraph PRE-CREATION 
  /trunk/pom.xml 1196926 
  /trunk/src/main/assembly/assembly.xml PRE-CREATION 
  /trunk/src/main/java/org/apache/giraph/GiraphRunner.java PRE-CREATION 
  /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1196926 

Diff: https://reviews.apache.org/r/2700/diff


Testing
---


Thanks,

Jakob

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142810#comment-13142810
 ] 

Avery Ching commented on GIRAPH-64:
---

Sorry, that was my bad.  Forgot to do a 'svn st -u' before the 'svn diff'.  
Thanks for the updated rb Jakob.  Btw, I'm a huge rb fan =).  If you specify 
the GIRAPH-64 in the 'Bugs:' section, it will automatically post to this issue.

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142797#comment-13142797
 ] 

Jakob Homan commented on GIRAPH-64:
---

ok, new rb request: https://reviews.apache.org/r/2700/diff/#index_header  
Apparently rb isn't smart enough to be able to use a patch generated from git.  
The mind boggles.

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: GIRAPH-64 Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching 
Crap.  I may have forgot to add some files.  Let me check when I get home
in an hour.  Sorry.

Sent from my iPhone

On Nov 2, 2011, at 7:43 PM, Jake Mannix  wrote:

  This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/

Does this have the whole diff?  Seems like it's missing stuff.  Only
has GraphMapper.java changes and pom.xml...


- Jake

On November 3rd, 2011, 2:29 a.m., Avery Ching wrote:
  Review request for giraph and Jake Mannix.
By Avery Ching.

*Updated 2011-11-03 02:29:54*
Description

This is filed on behalf of Jakob Homan.

  Testing

Don't know.

  *Bugs: * GIRAPH-64 
Diffs

   - http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml
   (1196921)
   -
   
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
   (1196921)

View Diff

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142790#comment-13142790
 ] 

Jakob Homan commented on GIRAPH-64:
---

Yeah, rb is missing the files.  My allergy to it grows.

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142786#comment-13142786
 ] 

jirapos...@reviews.apache.org commented on GIRAPH-64:
-


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/#review3021
---


Does this have the whole diff?  Seems like it's missing stuff.  Only has 
GraphMapper.java changes and pom.xml...

- Jake


On 2011-11-03 02:29:54, Avery Ching wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2691/
bq.  ---
bq.  
bq.  (Updated 2011-11-03 02:29:54)
bq.  
bq.  
bq.  Review request for giraph and Jake Mannix.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This is filed on behalf of Jakob Homan.
bq.  
bq.  
bq.  This addresses bug GIRAPH-64.
bq.  https://issues.apache.org/jira/browse/GIRAPH-64
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1196921 
bq.  
bq.  Diff: https://reviews.apache.org/r/2691/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Don't know.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Avery
bq.  
bq.



> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: GIRAPH-64 Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jake Mannix 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/#review3021
---


Does this have the whole diff?  Seems like it's missing stuff.  Only has 
GraphMapper.java changes and pom.xml...

- Jake


On 2011-11-03 02:29:54, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2691/
> ---
> 
> (Updated 2011-11-03 02:29:54)
> 
> 
> Review request for giraph and Jake Mannix.
> 
> 
> Summary
> ---
> 
> This is filed on behalf of Jakob Homan.
> 
> 
> This addresses bug GIRAPH-64.
> https://issues.apache.org/jira/browse/GIRAPH-64
> 
> 
> Diffs
> -
> 
>   http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
>  1196921 
> 
> Diff: https://reviews.apache.org/r/2691/diff
> 
> 
> Testing
> ---
> 
> Don't know.
> 
> 
> Thanks,
> 
> Avery
> 
>

[jira] [Commented] (GIRAPH-72) Running multiple Giraph jobs on the same cluster can lead to port collisions

2011-11-02 Thread Avery Ching (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142785#comment-13142785
 ] 

Avery Ching commented on GIRAPH-72:
---

Yes, this is a possible problem.  In the past, I've tried to grab all the map 
slots of a given task tracker via the appropriate memory configuration.  Right 
now, it's kind of nice to have the ports correspond to the task partition for 
debugability.  Would love to hear any other ideas.  

> Running multiple Giraph jobs on the same cluster can lead to port collisions
> 
>
> Key: GIRAPH-72
> URL: https://issues.apache.org/jira/browse/GIRAPH-72
> Project: Giraph
>  Issue Type: Bug
>  Components: lib, zookeeper
>Affects Versions: 0.70.0
> Environment: production hadoop cluster, in-process ZK.
>Reporter: Jake Mannix
>
> Had a Giraph mini-hackathon at work today, and lots of us launched 
> simultaneous test jobs at the same time, and often ran into the following 
> collision:
> --
> startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1
> 2-Nov-2011 23:40:08
> java.net.BindException: Problem binding to /:3 : 
> Address already in use
>   at org.apache.hadoop.ipc.Server.bind(Server.java:196)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:259)
>   at org.apache.hadoop.ipc.Server.(Server.java:1039)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:492)
>   at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
>   at 
> org.apache.giraph.comm.RPCCommunications.getRPCServer(RPCCommunications.java:99)
>   at 
> org.apache.giraph.comm.BasicRPCCommunications.(BasicRPCCommunications.java:362)
>   at org.apache.giraph.comm.RPCCommunications.(RPCCommunications.java:71)
>   at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:570)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>   at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:194)
>   ... 12 more
> 
> The job then simply hung.  What it should do, I'd imagine, is at a bare 
> minimum, catch this exception and allow the task to die quickly so it can get 
> retried on another machine, or better yet, allow for a command-line arg at 
> startup (and then passed into the Configuration) decide what ports to use.  
> Best yet, something automagic which allows multiple GraphMappers on the same 
> machine without manually picking ports (pick one at random and store it in 
> zookeeper?  but then what about the in-process zookeeper...) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142784#comment-13142784
 ] 

jirapos...@reviews.apache.org commented on GIRAPH-64:
-


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/
---

Review request for giraph and Jake Mannix.


Summary
---

This is filed on behalf of Jakob Homan.


This addresses bug GIRAPH-64.
https://issues.apache.org/jira/browse/GIRAPH-64


Diffs
-

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1196921 

Diff: https://reviews.apache.org/r/2691/diff


Testing
---

Don't know.


Thanks,

Avery



> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: GIRAPH-64 Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/
---

Review request for giraph and Jake Mannix.


Summary
---

This is filed on behalf of Jakob Homan.


This addresses bug GIRAPH-64.
https://issues.apache.org/jira/browse/GIRAPH-64


Diffs
-

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1196921 

Diff: https://reviews.apache.org/r/2691/diff


Testing
---

Don't know.


Thanks,

Avery

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142780#comment-13142780
 ] 

Avery Ching commented on GIRAPH-64:
---

I don't know how to make reviewboard do it automatically, but it's fairly easy. 
 I've done it a couple times.  I can do it for this one as an example.  Btw, 
Jake I'll let you review this one.  I think your expertise will be useful.

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (GIRAPH-72) Running multiple Giraph jobs on the same cluster can lead to port collisions

2011-11-02 Thread Jake Mannix (Created) (JIRA) 
Running multiple Giraph jobs on the same cluster can lead to port collisions


 Key: GIRAPH-72
 URL: https://issues.apache.org/jira/browse/GIRAPH-72
 Project: Giraph
  Issue Type: Bug
  Components: lib, zookeeper
Affects Versions: 0.70.0
 Environment: production hadoop cluster, in-process ZK.
Reporter: Jake Mannix


Had a Giraph mini-hackathon at work today, and lots of us launched simultaneous 
test jobs at the same time, and often ran into the following collision:

--
startSuperstep: WORKER_ONLY - Attempt=0, Superstep=-1
2-Nov-2011 23:40:08

java.net.BindException: Problem binding to /:3 : Address 
already in use
at org.apache.hadoop.ipc.Server.bind(Server.java:196)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:259)
at org.apache.hadoop.ipc.Server.(Server.java:1039)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:492)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
at 
org.apache.giraph.comm.RPCCommunications.getRPCServer(RPCCommunications.java:99)
at 
org.apache.giraph.comm.BasicRPCCommunications.(BasicRPCCommunications.java:362)
at org.apache.giraph.comm.RPCCommunications.(RPCCommunications.java:71)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:570)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:194)
... 12 more


The job then simply hung.  What it should do, I'd imagine, is at a bare 
minimum, catch this exception and allow the task to die quickly so it can get 
retried on another machine, or better yet, allow for a command-line arg at 
startup (and then passed into the Configuration) decide what ports to use.  
Best yet, something automagic which allows multiple GraphMappers on the same 
machine without manually picking ports (pick one at random and store it in 
zookeeper?  but then what about the in-process zookeeper...) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142615#comment-13142615
 ] 

Jakob Homan commented on GIRAPH-36:
---

bq. Can GIRAPH-47 be a start for what you have in mind?
Yeah, that's close to what I'm hoping for.  I'll comment there. 

bq. I can certainly imagine this being very nice and clean.
Inheritance is going to be brittle over the long term, so the cleaner we can 
make this now, the easier it will be to attract a user base confident in 
investing in the new platform.  This is particularly true given how much we're 
at the mercy of Java's poor generic system; users already have to grok things 
like DoubleDoubleFloat in type names.

> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
> VertexRange extends ArrayList, and there are other places where the 
> assumption is that vertex classes are either Vertex, or at least 
> MutableVertex.
> Let's make sure the internal APIs allow for BasicVertex to be the base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142608#comment-13142608
 ] 

Jakob Homan commented on GIRAPH-64:
---

Not that I know of.  I'm allergic to rb, but I've never seen one.  In Hive land 
we have to manually create an instance for each issue.  I just tried to create 
one for this and, while Giraph exists as a target, I get a 500 exception 
("something broke!" - gee, that's helpful).

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jake Mannix (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142593#comment-13142593
 ] 

Jake Mannix commented on GIRAPH-64:
---

I'd be happy to review this too (I wrote the original form of bin/mahout's 
MahoutDriver program, although it's gone and evolved over time from there) is 
there a way to automagically have ReviewBoard create a review for this patch, 
so we can see it in a GUI?  My brain has atrophied greatly as IDE's have 
advanced. ;P

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Claudio Martella 
Hi Jake,

yep, you basically understand it correctly, it is indeed something
like a SSSP, but I'm starting from many vertices at the same time (so
i'd call it a grammar-based ASSP).

To make it more clear, the current code looks like:

@Override
public void compute(Iterator messages)
throws IOException {

if (getSuperstep() == 0 && isSource()) {
processMessage(new Message(this.query, new ResultSet()));
} else {
while (messages.hasNext()) {
processMessage(messages.next());
}
}

voteToHalt();
}

as you can see I have used a hack and saved the query as VertexValue
(so I at least saved parsing it multiple times from String).

With what I have in mind, I could remove the first if-condition completely.
Also, as not all the vertices are starting vertices, I could obtain
the complete list of starting vertices by running a filter based on
the first element of my stack/query. Suppose my query asks for all the
actors who have acted in a movie which took place in USA. (actor -
acted_in -> movie - takes_place -> USA), I could set as a starting
vertices only those vertices that have an outgoing edge with label
"acted_in". That would save quite a lot. As we didn't have an API to
do this yet, I still haven't figured out how to do this precisely, but
i guess this could be done in the VertexReader.

On Wed, Nov 2, 2011 at 4:51 PM, Jake Mannix  wrote:
> On Wed, Nov 2, 2011 at 5:07 AM, Claudio Martella > wrote:
>>
>> Not each vertex is a starting vertex for the path traversals, so it
>> could be nice if at the first superstep the starting vertices could
>> already have the messages in their inbox. This way the message
>> processing (and path traversal processing with it) could be fully
>> transparent (vertices just have to iterate throw messages without
>> caring about parsing, start vertices etc etc).
>>
>> Currently, at first superstep, each vertex has to get che query from
>> the conf, parse it into a stack and discard it afterwards, as soon as
>> it discovers it is not a starting vertex for a path traversal.
>> Outside, I know who's a starting vertex and could just set the inbox
>> for those vertices accordingly.
>>
>
> Let me understand what the situation looks like, by basically repeating
> this back to you: in your case, the set of queries which are to be run
> over your graph need to be read/parsed at startup and "given" to the
> starting node who is going to be responsible for initiating their
> propagation
> across the graph, right?
>
> For one thing, storing this batch set of queries in the Configuration
> doesn't seem very scalable - having that live on-disk and be read
> by the VertexReader does seem to make sense, although I wonder how
> easy it would be to coerce your serialized on-disk format to be nice
> and re-use the unchanging graph for multiple queries?  Maybe a
> MultipleInputFormats kind of thing would be needed?
>
> So how is this different from SimpleShortestPathsVertex, in the way
> that one vertex is "special"?  The compute() method of that vertex
> uses the knowledge of whether it is the source or not to set its "distance"
> from the origin, and only when that distance is > the current min does
> it send messages out.
>
> I could see how algorithms like this could possibly be sped up if we
> didn't need to iterate over *all* of the graph on each superstep, as the
> first step only needs to send messages to the vertices directly
> connected to the origin, then only it's 2nd degree, etc.  If we were
> only to process the vertices which had messages to send, for example,
> and iterate over those in a sparse fashion, this algorithm could be
> sped up considerably.
>
> I guess you're saying that similarly, not only if the messages were
> directly associated with their starting nodes, but if during the superstep
> we could only iterate over the nodes with messages, we could be much
> faster.
>
> It seems there might be a bunch of algorithms which would work this
> way, and I wonder if it would be a good idea to modify the system to
> have GraphMapper check to see if the vertex class "isSparse()" in
> some sense, which would mean that in the superstep walk:
>
>            for (Map.Entry> entry :
>                serviceWorker.getVertexRangeMap().entrySet()) {
> // ...
>                for (BasicVertex vertex :
>                        entry.getValue().getVertexMap().values()) {
> // ...
>                    if (!vertex.isHalted()) {
>                        Iterator vertexMsgIt =
>                            vertex.getMsgList().iterator();
>                         context.progress();
>                        vertex.compute(vertexMsgIt);
>                    }
>
> we instead would iterate over a sparse data structure which
> only contains the vertices to be processed (this data structure
> would of course change from superstep to superstep as the
> messages propagate across the graph).  This could be a pret

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Jake Mannix (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142589#comment-13142589
 ] 

Jake Mannix commented on GIRAPH-36:
---

It's tricky, I think.  The original Vertex class was exactly meant to hide (via 
private) some of the implementation details of state, yet leave userland APIs 
which the user is meant to implement.  But then it turned out that the most 
generic way to hold onto the destination vertices and edge weights is not 
always the most efficient (hence: primitive-specific subclasses).

But maybe you're just saying that we should pull out most of this 
"implementation specific" state into other objects, decompose Vertex a bit, and 
have users be able to pluggably implement the parts which need to be made 
special, and leave as generic those which are generic?

Some stuff we already have done: GraphState encapsulates all global state 
(current superstep, number of global edges + vertices, etc).  Local state could 
be similarly pulled off into a couple of different data structures (edges with 
weights, messages).

I can certainly imagine this being very nice and clean.

Open a ticket and describe some thoughts on how it could look in practice?  
Sounds like it could be another pretty significant refactoring, so let's do it 
sooner rather than later!

> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
> VertexRange extends ArrayList, and there are other places where the 
> assumption is that vertex classes are either Vertex, or at least 
> MutableVertex.
> Let's make sure the internal APIs allow for BasicVertex to be the base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Claudio Martella (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142581#comment-13142581
 ] 

Claudio Martella commented on GIRAPH-36:


Hi Jakob,

as a matter of fact, I believe, part of your proposal is addressed by 
GIRAPH-47. 
Now that Jake has committed this work, I'll sync and complete that patch to the 
current trunk (I was out of office for conference the whole last week).

Can GIRAPH-47 be a start for what you have in mind?

> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
> VertexRange extends ArrayList, and there are other places where the 
> assumption is that vertex classes are either Vertex, or at least 
> MutableVertex.
> Let's make sure the internal APIs allow for BasicVertex to be the base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-71) SequenceFileVertexInputFormat missing license header; rat fails

2011-11-02 Thread Hudson (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142578#comment-13142578
 ] 

Hudson commented on GIRAPH-71:
--

Integrated in Giraph-trunk-Commit #23 (See 
[https://builds.apache.org/job/Giraph-trunk-Commit/23/])
GIRAPH-71. SequenceFileVertexInputFormat missing license header; rat fails.

jghoman : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1196814
Files : 
* /incubator/giraph/trunk/CHANGELOG
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/SequenceFileVertexInputFormat.java


> SequenceFileVertexInputFormat missing license header; rat fails
> ---
>
> Key: GIRAPH-71
> URL: https://issues.apache.org/jira/browse/GIRAPH-71
> Project: Giraph
>  Issue Type: Bug
>  Components: lib
>Affects Versions: 0.70.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.70.0
>
> Attachments: GIRAPH-71.patch
>
>
> SequenceFileVertexInputFormat needs the Apache license header.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (GIRAPH-71) SequenceFileVertexInputFormat missing license header; rat fails

2011-11-02 Thread Jakob Homan (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/GIRAPH-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakob Homan resolved GIRAPH-71.
---

Resolution: Fixed

I've committed this.  Resolving as fixed.

> SequenceFileVertexInputFormat missing license header; rat fails
> ---
>
> Key: GIRAPH-71
> URL: https://issues.apache.org/jira/browse/GIRAPH-71
> Project: Giraph
>  Issue Type: Bug
>  Components: lib
>Affects Versions: 0.70.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.70.0
>
> Attachments: GIRAPH-71.patch
>
>
> SequenceFileVertexInputFormat needs the Apache license header.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142573#comment-13142573
 ] 

Jakob Homan commented on GIRAPH-36:
---

Great work, Jake. One thing I notice while reading through Vertex is muddling 
of the state the Vertex is responsible for on a per-application basis and the 
state Giraph manages for the vertex.  I think we may being ill-served by 
inheritance here and should instead rely on composition to hold this state.  
I'm thinking that messages in/out, edge state and mutation, facilities for 
sending messages, current superstep, etc. Would it be better in the long term 
to move these out to a context object (ala MR)?  This would simplify Vertex 
significantly, make it much easier to test (by mocking out the dependency) and 
future proof the evolution of Vertex as there will be fewer moving parts to 
keep track of.

Does changing compute and {pre|post}{Superstep|Application} took their external 
state as parameters with sound like a good approach to explore?

> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
> VertexRange extends ArrayList, and there are other places where the 
> assumption is that vertex classes are either Vertex, or at least 
> MutableVertex.
> Let's make sure the internal APIs allow for BasicVertex to be the base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-71) SequenceFileVertexInputFormat missing license header; rat fails

2011-11-02 Thread Jake Mannix (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142569#comment-13142569
 ] 

Jake Mannix commented on GIRAPH-71:
---

+1 from me, not sure why it passed for me, I was sure I'd run the full "mvn 
clean install" which runs rat too, but then when I tried re-cloning again this 
morning, it failed.  Thanks Jakob!

> SequenceFileVertexInputFormat missing license header; rat fails
> ---
>
> Key: GIRAPH-71
> URL: https://issues.apache.org/jira/browse/GIRAPH-71
> Project: Giraph
>  Issue Type: Bug
>  Components: lib
>Affects Versions: 0.70.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.70.0
>
> Attachments: GIRAPH-71.patch
>
>
> SequenceFileVertexInputFormat needs the Apache license header.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (GIRAPH-71) SequenceFileVertexInputFormat missing license header; rat fails

2011-11-02 Thread Jakob Homan (Updated) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/GIRAPH-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakob Homan updated GIRAPH-71:
--

Attachment: GIRAPH-71.patch

quick patch.

> SequenceFileVertexInputFormat missing license header; rat fails
> ---
>
> Key: GIRAPH-71
> URL: https://issues.apache.org/jira/browse/GIRAPH-71
> Project: Giraph
>  Issue Type: Bug
>  Components: lib
>Affects Versions: 0.70.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.70.0
>
> Attachments: GIRAPH-71.patch
>
>
> SequenceFileVertexInputFormat needs the Apache license header.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (GIRAPH-71) SequenceFileVertexInputFormat missing license header; rat fails

2011-11-02 Thread Jakob Homan (Created) (JIRA) 
SequenceFileVertexInputFormat missing license header; rat fails
---

 Key: GIRAPH-71
 URL: https://issues.apache.org/jira/browse/GIRAPH-71
 Project: Giraph
  Issue Type: Bug
  Components: lib
Affects Versions: 0.70.0
Reporter: Jakob Homan
Assignee: Jakob Homan
 Fix For: 0.70.0


SequenceFileVertexInputFormat needs the Apache license header.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Jakob Homan (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142514#comment-13142514
 ] 

Jakob Homan commented on GIRAPH-64:
---

bq. Anyone want to take a crack at reviewing this? If not, I'll get to it 
tomorrow night.
I'll be gone for the next few days, so I'm hoping it can get reviewed and 
committed.  I think it's an important step forward.  Once this is in, we 
generalize the provided examples (pagerank, shortest path, etc.) to work with 
any inputformat/outputformat and make Giraph immediately useful to anyone who 
downloads it and wants to play with a reasonably stored graph.  

I've verified it still merges cleanly after GIRAPH-36.

> Create VertexRunner to make it easier to run users' computations
> 
>
> Key: GIRAPH-64
> URL: https://issues.apache.org/jira/browse/GIRAPH-64
> Project: Giraph
>  Issue Type: New Feature
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Attachments: GIRAPH-64.patch
>
>
> Currently, if a user wants to implement a Giraph algorithm by extending 
> {{Vertex}} they must also write all the boilerplate around the {{Tool}} 
> interface and bundle it with the Giraph jar (or get Giraph on the classpath 
> and playing nice with the implementation).  For example, what is included in 
> the PageRankBenchmark and what Kohei has done: 
> https://github.com/smly/java-Giraph-LabelPropagation  It would be better if 
> we had perhaps a Vertex implementation to be subclassed that already had all 
> the standard Tooling included such that all one had to run would be (assuming 
> the Giraph jar was already on the classpath):
> {noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o 
> jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble 
> -of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat} 
> This wouldn't work with every algorithm, but would be useful in a large 
> number of cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (GIRAPH-28) Introduce new primitive-specific MutableVertex subclasses

2011-11-02 Thread Jake Mannix (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/GIRAPH-28?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jake Mannix resolved GIRAPH-28.
---

   Resolution: Fixed
Fix Version/s: 0.70.0

This was included in GIRAPH-36

> Introduce new primitive-specific MutableVertex subclasses
> -
>
> Key: GIRAPH-28
> URL: https://issues.apache.org/jira/browse/GIRAPH-28
> Project: Giraph
>  Issue Type: New Feature
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
> Fix For: 0.70.0
>
> Attachments: GIRAPH-28.diff, GIRAPH-28.diff, GIRAPH-28.diff
>
>
> As discussed on the list, 
> MutableVertex (for 
> example) could be highly optimized in its memory footprint if the vertex and 
> edge data were held in a form which minimized Java object usage.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Jake Mannix 
On Wed, Nov 2, 2011 at 5:07 AM, Claudio Martella  wrote:
>
> Not each vertex is a starting vertex for the path traversals, so it
> could be nice if at the first superstep the starting vertices could
> already have the messages in their inbox. This way the message
> processing (and path traversal processing with it) could be fully
> transparent (vertices just have to iterate throw messages without
> caring about parsing, start vertices etc etc).
>
> Currently, at first superstep, each vertex has to get che query from
> the conf, parse it into a stack and discard it afterwards, as soon as
> it discovers it is not a starting vertex for a path traversal.
> Outside, I know who's a starting vertex and could just set the inbox
> for those vertices accordingly.
>

Let me understand what the situation looks like, by basically repeating
this back to you: in your case, the set of queries which are to be run
over your graph need to be read/parsed at startup and "given" to the
starting node who is going to be responsible for initiating their
propagation
across the graph, right?

For one thing, storing this batch set of queries in the Configuration
doesn't seem very scalable - having that live on-disk and be read
by the VertexReader does seem to make sense, although I wonder how
easy it would be to coerce your serialized on-disk format to be nice
and re-use the unchanging graph for multiple queries?  Maybe a
MultipleInputFormats kind of thing would be needed?

So how is this different from SimpleShortestPathsVertex, in the way
that one vertex is "special"?  The compute() method of that vertex
uses the knowledge of whether it is the source or not to set its "distance"
from the origin, and only when that distance is > the current min does
it send messages out.

I could see how algorithms like this could possibly be sped up if we
didn't need to iterate over *all* of the graph on each superstep, as the
first step only needs to send messages to the vertices directly
connected to the origin, then only it's 2nd degree, etc.  If we were
only to process the vertices which had messages to send, for example,
and iterate over those in a sparse fashion, this algorithm could be
sped up considerably.

I guess you're saying that similarly, not only if the messages were
directly associated with their starting nodes, but if during the superstep
we could only iterate over the nodes with messages, we could be much
faster.

It seems there might be a bunch of algorithms which would work this
way, and I wonder if it would be a good idea to modify the system to
have GraphMapper check to see if the vertex class "isSparse()" in
some sense, which would mean that in the superstep walk:

for (Map.Entry> entry :
serviceWorker.getVertexRangeMap().entrySet()) {
// ...
for (BasicVertex vertex :
entry.getValue().getVertexMap().values()) {
// ...
if (!vertex.isHalted()) {
Iterator vertexMsgIt =
vertex.getMsgList().iterator();
 context.progress();
vertex.compute(vertexMsgIt);
}

we instead would iterate over a sparse data structure which
only contains the vertices to be processed (this data structure
would of course change from superstep to superstep as the
messages propagate across the graph).  This could be a pretty
significant speedup a lot of the time in practice, as it would
mean that iterations per superstep would scale in complexity
as O( |currently visible part of the graph| ) as opposed to
O( | entire graph| ), even if most of the iteration is just a bunch
of in-memory boolean and size checks.

Not sure if this is exactly what you're talking about, however.

  -jake

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Hudson (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142221#comment-13142221
 ] 

Hudson commented on GIRAPH-36:
--

Integrated in Giraph-trunk-Commit #22 (See 
[https://builds.apache.org/job/Giraph-trunk-Commit/22/])
GIRAPH-36: Ensure that subclassing BasicVertex is possible by user
  apps. (jake.mannix via aching)

aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1196639
Files : 
* /incubator/giraph/trunk/CHANGELOG
* /incubator/giraph/trunk/pom.xml
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PseudoRandomVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexList.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerCommunications.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexReader.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertexResolver.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Edge.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexInputFormat.java
* /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexReader.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/AdjacencyListTextVertexOutputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/AdjacencyListVertexReader.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/JsonBase64VertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/LongDoubleDoubleAdjacencyListVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/SequenceFileVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/TextDoubleDoubleAdjacencyListVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/TextVertexInputFormat.java
* 
/incubator/giraph/trunk/src/main/java/org/apache/giraph/zk/ZooKeeperManager.java
* /incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* 
/incubator/giraph/trunk/src/test/java/org/apache/giraph/lib/TestAdjacencyListTextVertexOutputFormat.java
* 
/incubator/giraph/trunk/src/test/java/org/apache/giraph/lib/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java
* 
/incubator/giraph/trunk/src/test/java/org/apache/giraph/lib/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java


> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
>

Re: [jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Claudio Martella 
Great, so I can I base GIRAPH-10 and GIRAPH-47 after this one!

About my use case:

I'm implementing a batch path query engine where the queries are
XPath-like queries. Each query is a stack of conditions, each element
assigned to a depth of the path traversal. A message is a stack.

Not each vertex is a starting vertex for the path traversals, so it
could be nice if at the first superstep the starting vertices could
already have the messages in their inbox. This way the message
processing (and path traversal processing with it) could be fully
transparent (vertices just have to iterate throw messages without
caring about parsing, start vertices etc etc).

Currently, at first superstep, each vertex has to get che query from
the conf, parse it into a stack and discard it afterwards, as soon as
it discovers it is not a starting vertex for a path traversal.
Outside, I know who's a starting vertex and could just set the inbox
for those vertices accordingly.

Hope this helps,
Claudio

On Wed, Nov 2, 2011 at 9:14 AM, Jake Mannix (Commented) (JIRA)
 wrote:
>
>    [ 
> https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141994#comment-13141994
>  ]
>
> Jake Mannix commented on GIRAPH-36:
> ---
>
> Excellent, looks good to me.
> +1 for committing we can iterate from here is the main point.  The sooner 
> this gets in, the fewer other patches out there get broken. ;)
>
>> Ensure that subclassing BasicVertex is possible by user apps
>> 
>>
>>                 Key: GIRAPH-36
>>                 URL: https://issues.apache.org/jira/browse/GIRAPH-36
>>             Project: Giraph
>>          Issue Type: Improvement
>>          Components: graph
>>    Affects Versions: 0.70.0
>>            Reporter: Jake Mannix
>>            Assignee: Jake Mannix
>>            Priority: Blocker
>>             Fix For: 0.70.0
>>
>>         Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
>> GIRAPH-36.diff.warnings
>>
>>
>> Original assumptions in Giraph were that all users would subclass Vertex 
>> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
>> have application specific data structures (ie. not a TreeMap>) 
>> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
>> VertexRange extends ArrayList, and there are other places where the 
>> assumption is that vertex classes are either Vertex, or at least 
>> MutableVertex.
>> Let's make sure the internal APIs allow for BasicVertex to be the base class.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA 
> administrators: 
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>



-- 
    Claudio Martella
    claudio.marte...@gmail.com

[jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-11-02 Thread Jake Mannix (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141994#comment-13141994
 ] 

Jake Mannix commented on GIRAPH-36:
---

Excellent, looks good to me.
+1 for committing we can iterate from here is the main point.  The sooner this 
gets in, the fewer other patches out there get broken. ;)

> Ensure that subclassing BasicVertex is possible by user apps
> 
>
> Key: GIRAPH-36
> URL: https://issues.apache.org/jira/browse/GIRAPH-36
> Project: Giraph
>  Issue Type: Improvement
>  Components: graph
>Affects Versions: 0.70.0
>Reporter: Jake Mannix
>Assignee: Jake Mannix
>Priority: Blocker
> Fix For: 0.70.0
>
> Attachments: GIRAPH-36.diff, GIRAPH-36.diff, GIRAPH-36.diff, 
> GIRAPH-36.diff.warnings
>
>
> Original assumptions in Giraph were that all users would subclass Vertex 
> (which extended MutableVertex extended BasicVertex).  Classes which wish to 
> have application specific data structures (ie. not a TreeMap>) 
> may need to extend either MutableVertex or BasicVertex.  Unfortunately 
> VertexRange extends ArrayList, and there are other places where the 
> assumption is that vertex classes are either Vertex, or at least 
> MutableVertex.
> Let's make sure the internal APIs allow for BasicVertex to be the base class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira