from:"Avery Ching"

Re: Review Request: Improve concurrency of putMsg / putMsgList

2012-04-24 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4852/#review7185
---



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java


Bo, I'm a little leery about converting the List and ArrayList to 
LinkedList and ConcurrentLinkedList.  I believe that linked list's will use 
more memory than the array list due to the double links (forward and backward). 
 Also, is ConcurrentLinkedList supposted to outperform a synchronized 
ArrayList?  I haven't seen much on that.

The concurrenthashmap changes look good.


- Avery


On 2012-04-24 06:11:38, Bo Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4852/
> ---
> 
> (Updated 2012-04-24 06:11:38)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Use ConcurrentHashMap and ConcurrentLinkedQueue to allow concurrent assess to 
> message map. The concurrencyLevel of ConcurrentHashMap uses the default 
> value. There may be some performance gain by tuning this value.
> 
> 
> This addresses bug GIRAPH-185.
> https://issues.apache.org/jira/browse/GIRAPH-185
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
>  1328747 
> 
> Diff: https://reviews.apache.org/r/4852/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bo
> 
>

Re: Review Request: Improve concurrency of putMsg / putMsgList

2012-04-24 Thread Avery Ching



> On 2012-04-24 20:53:33, Avery Ching wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java,
> >  lines 776-777
> > <https://reviews.apache.org/r/4852/diff/1/?file=104059#file104059line776>
> >
> > Bo, I'm a little leery about converting the List and ArrayList to 
> > LinkedList and ConcurrentLinkedList.  I believe that linked list's will use 
> > more memory than the array list due to the double links (forward and 
> > backward).  Also, is ConcurrentLinkedList supposted to outperform a 
> > synchronized ArrayList?  I haven't seen much on that.
> > 
> > The concurrenthashmap changes look good.
> 
> Bo Wang wrote:
> Avery, thanks for the comments. I just measured the sizes of these 
> classes and below are an estimation. 
> 
> java.util.ArrayList: 149 bytes
> java.util.LinkedList: 101 bytes
> java.util.concurrent.ConcurrentLinkedQueue: 118 bytes
> 
> The tool I was using is a program from the link below.
> http://www.javapractices.com/topic/TopicAction.do?Id=83
> 
> In terms of performance, here is a benchmark.
> 
> http://www.javacodegeeks.com/2010/09/java-best-practices-queue-battle-and.html
> 
> In its test #1 (adding element), ConcurrentLinkedQueue performed slightly 
> better than LinkedList. In test #3 (iterator), LinkedList outperformed 
> ConcurrentLinkedQueue. I think the most time consuming part is add, while 
> iteration is also heavily used but no concurrent accesses. 
> 
>

Thanks for the response Bo.

Those numbers are for the empty data structures I'm assuming.  I was referring 
to the incremental cost of adding elements (messages) to the data structures.  
The performance isn't a a concern to me (unless we call size() somewhere).


- Avery


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4852/#review7185
---


On 2012-04-24 06:11:38, Bo Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4852/
> ---
> 
> (Updated 2012-04-24 06:11:38)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Use ConcurrentHashMap and ConcurrentLinkedQueue to allow concurrent assess to 
> message map. The concurrencyLevel of ConcurrentHashMap uses the default 
> value. There may be some performance gain by tuning this value.
> 
> 
> This addresses bug GIRAPH-185.
> https://issues.apache.org/jira/browse/GIRAPH-185
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
>  1328747 
> 
> Diff: https://reviews.apache.org/r/4852/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bo
> 
>

Re: Review Request: Improve concurrency of putMsg / putMsgList

2012-04-24 Thread Avery Ching



> On 2012-04-24 20:53:33, Avery Ching wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java,
> >  lines 776-777
> > <https://reviews.apache.org/r/4852/diff/1/?file=104059#file104059line776>
> >
> > Bo, I'm a little leery about converting the List and ArrayList to 
> > LinkedList and ConcurrentLinkedList.  I believe that linked list's will use 
> > more memory than the array list due to the double links (forward and 
> > backward).  Also, is ConcurrentLinkedList supposted to outperform a 
> > synchronized ArrayList?  I haven't seen much on that.
> > 
> > The concurrenthashmap changes look good.
> 
> Bo Wang wrote:
> Avery, thanks for the comments. I just measured the sizes of these 
> classes and below are an estimation. 
> 
> java.util.ArrayList: 149 bytes
> java.util.LinkedList: 101 bytes
> java.util.concurrent.ConcurrentLinkedQueue: 118 bytes
> 
> The tool I was using is a program from the link below.
> http://www.javapractices.com/topic/TopicAction.do?Id=83
> 
> In terms of performance, here is a benchmark.
> 
> http://www.javacodegeeks.com/2010/09/java-best-practices-queue-battle-and.html
> 
> In its test #1 (adding element), ConcurrentLinkedQueue performed slightly 
> better than LinkedList. In test #3 (iterator), LinkedList outperformed 
> ConcurrentLinkedQueue. I think the most time consuming part is add, while 
> iteration is also heavily used but no concurrent accesses. 
> 
>
> 
> Avery Ching wrote:
> Thanks for the response Bo.
> 
> Those numbers are for the empty data structures I'm assuming.  I was 
> referring to the incremental cost of adding elements (messages) to the data 
> structures.  The performance isn't a a concern to me (unless we call size() 
> somewhere).

By the incremental cost, I mean the memory cost, sorry.


- Avery


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4852/#review7185
---


On 2012-04-24 06:11:38, Bo Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4852/
> ---
> 
> (Updated 2012-04-24 06:11:38)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Use ConcurrentHashMap and ConcurrentLinkedQueue to allow concurrent assess to 
> message map. The concurrencyLevel of ConcurrentHashMap uses the default 
> value. There may be some performance gain by tuning this value.
> 
> 
> This addresses bug GIRAPH-185.
> https://issues.apache.org/jira/browse/GIRAPH-185
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
>  1328747 
> 
> Diff: https://reviews.apache.org/r/4852/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bo
> 
>

Re: Review Request: HBase/Accumulo Input and Output formats (on behalf of Brian)

2012-04-30 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4801/
---

(Updated 2012-04-30 23:54:55.758151)


Review request for giraph.


Changes
---

Update of Brian's 153.2.


Summary
---

Brian's patch for GIRAPH-153.


This addresses bug GIRAPH-153.
https://issues.apache.org/jira/browse/GIRAPH-153


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/LICENSE.txt
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/license-header.txt
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/pom.xml
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/assembly/compile.xml
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexOutputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/package-info.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/HBaseVertexInputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/HBaseVertexOutputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/package-info.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/BspCase.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/accumulo/TestAccumuloVertexFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/accumulo/edgemarker/AccumuloEdgeInputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/accumulo/edgemarker/AccumuloEdgeOutputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/hbase/TestHBaseRootMarkerVertextFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/hbase/edgemarker/TableEdgeInputFormat.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/hbase/edgemarker/TableEdgeOutputFormat.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/4801/diff


Testing
---


Thanks,

Avery

Re: Review Request: HBase/Accumulo Input and Output formats (on behalf of Brian)

2012-04-30 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4801/#review7404
---


Hi Brian, the patch applies nicely, but it filled with duplicates.  Also there 
are some javadoc indentation fixes to make.  I just gave a couple of examples.  
Can you please fix this and resubmit?  Thanks!


http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/LICENSE.txt
<https://reviews.apache.org/r/4801/#comment16328>

This license is duplicated several times.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/license-header.txt
<https://reviews.apache.org/r/4801/#comment16329>

This license is duplicated several times.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/assembly/compile.xml
<https://reviews.apache.org/r/4801/#comment16330>

More duplication.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16331>

Please fix indentation.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16332>

Please fix indentation.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16333>

Please fix indentation.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16334>

Please fix indentation.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16335>

Code duplication.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
<https://reviews.apache.org/r/4801/#comment16336>

Extra *


- Avery


On 2012-04-30 23:54:55, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4801/
> ---
> 
> (Updated 2012-04-30 23:54:55)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Brian's patch for GIRAPH-153.
> 
> 
> This addresses bug GIRAPH-153.
> https://issues.apache.org/jira/browse/GIRAPH-153
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/LICENSE.txt
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/license-header.txt
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/pom.xml
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/assembly/compile.xml
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexInputFormat.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/AccumuloVertexOutputFormat.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/accumulo/package-info.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/HBaseVertexInputFormat.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/HBaseVertexOutputFormat.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/format/hbase/package-info.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/BspCase.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/format/accum

Re: [DISCUSS] Giraph graduation resolution

2012-05-04 Thread Avery Ching


+1, sounds good to me.  I also would like to have a rotating PMC chair.

Avery

On 5/4/12 10:26 AM, Jakob Homan wrote:

Both suggestions sound reasonable to me.  +1 on the current resolution.

On Fri, May 4, 2012 at 10:23 AM, Owen O'Malley  wrote:

On Fri, May 4, 2012 at 10:09 AM, Jakob Homan  wrote:

Looks good.  I might suggest adding language to rotate the PMC chair
annually to spread the responsibility around a bit and increase our
Bus Number.  Also, I was hoping to have seen Christian a bit more
during incubation...

Since this resolution is a one-off, I would suggest putting the anual
rotation in the bylaws that will be part of the project's permanent
website. That will make it more visible and easier for the project to
change it itself.

I agree that Christian hasn't been involved while it is in the
incubator. On the other hand, he was heavily involved before it came
to Apache. Looking at the svn logs, the number of commits (not
contributions) per user is:

  148 aching
  47 jghoman
  37 ckunz
  11 claudio
   7 exg
   5 ssc
   4 hyunsik
   3 omalley
   1 kunzchr
   1 jmannix
   1 ekoontz
   1 asuresh

So Christian only has 1 commit at Apache, but he has 37 prior. Given
that level of involvement, I'd rather put him on the Giraph PMC and
let him go emeritis in a few months. What do others think?

-- Owen

Review Request: Implemented a netty client/server protocol a a faster alternative to Hadoop RPC (3x improvement)

2012-05-09 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5074/
---

Review request for giraph.


Summary
---

* Implemented a request/response protocol with netty as a NettyClient and 
NettyServer.  There is a NettyClientWorker and NettyClientServer that 
implements WorkerClient and WorkerServer, respectively.  Netty is a lot faster 
since it's non-blocking and we can interleave computation and communication as 
opposed to Hadoop RPC (blocking).
* The netty server implementation uses concurrent hash maps to improved 
concurrency instead of synchronized blocks around maps.
* By default netty is used, but Hadoop RPC can be used with 
-Dgiraph.useNetty=false
* Changed the class hierarchy of ServerInterface to WorkerClientServer 
(WorkerClient and WorkerServer) to support a request/response protocol instead 
of just RPC
* In netty, the messages/mutations are gathered by partition and send out as a 
partition's worth of messages/mutations
* Added two new test classes (RequestTest.java and ConnectionTest.java) to test 
all requests and check netty connections.
* PageRankBenchmark uses EdgeListVertex as a default


This addresses bug GIRAPH-37.
https://issues.apache.org/jira/browse/GIRAPH-37


Diffs
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/utils/MockUtils.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/examples/SimpleShortestPathVertexTest.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/comm/ConnectionTest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexMutations.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WritableRequest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClientServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerCommunications.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClient.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ServerInterface.java
 1332888 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ServerData.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/SendVertexRequest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/SendPartitionMutationsRequest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/SendPartitionMessagesRequest.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/SendMutationsCache.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/SendMessageCache.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/RequestServerHandler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ResponseClientHandler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/RequestRegistry.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/RequestEncoder.ja

Re: Review Request: Implemented a netty client/server protocol a a faster alternative to Hadoop RPC (3x improvement)

2012-05-09 Thread Avery Ching



> On 2012-05-09 10:10:46, Sebastian Schelter wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java,
> >  line 1465
> > <https://reviews.apache.org/r/5074/diff/2/?file=108120#file108120line1465>
> >
> > I don't like it that a collection is changed outside of the class that 
> > owns it. 
> > 
> > This makes code hard to read and debug. We should rather introduce a 
> > method for this in the class that owns this map to have all mutations in 
> > one place.

Good point, it's a little heard to understand.  Since this is a Map, we can do 
as you suggested, keep it in a class and then add a method to do the clear().  
We can even add calls to do the methods that iterate over the map as well to 
not have to do any synchronization outside of the map.  I'll do this for all 
our synchronized objects in the next patch if that's okay with you (the current 
code does this as well).  It will be a somewhat medium sized change.


- Avery


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5074/#review7728
-------


On 2012-05-09 09:22:36, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5074/
> ---
> 
> (Updated 2012-05-09 09:22:36)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> * Implemented a request/response protocol with netty as a NettyClient and 
> NettyServer.  There is a NettyClientWorker and NettyClientServer that 
> implements WorkerClient and WorkerServer, respectively.  Netty is a lot 
> faster since it's non-blocking and we can interleave computation and 
> communication as opposed to Hadoop RPC (blocking).
> * The netty server implementation uses concurrent hash maps to improved 
> concurrency instead of synchronized blocks around maps.
> * By default netty is used, but Hadoop RPC can be used with 
> -Dgiraph.useNetty=false
> * Changed the class hierarchy of ServerInterface to WorkerClientServer 
> (WorkerClient and WorkerServer) to support a request/response protocol 
> instead of just RPC
> * In netty, the messages/mutations are gathered by partition and send out as 
> a partition's worth of messages/mutations
> * Added two new test classes (RequestTest.java and ConnectionTest.java) to 
> test all requests and check netty connections.
> * PageRankBenchmark uses EdgeListVertex as a default
> 
> 
> This addresses bug GIRAPH-37.
> https://issues.apache.org/jira/browse/GIRAPH-37
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/utils/MockUtils.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/examples/SimpleShortestPathVertexTest.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/comm/ConnectionTest.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexMutations.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WritableRequest.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
>  1332888 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerClientServer.java
>  PRE-CREATION

Re: Review Request: GIRAPH-20 Move temporary test files from the project directory

2012-05-09 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5077/#review7745
---


Hey, Sebastian, overall looks good.  If no one else gets to it, I'll finish 
this review tonight.

- Avery


On 2012-05-09 11:37:47, Sebastian Schelter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5077/
> ---
> 
> (Updated 2012-05-09 11:37:47)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> All temporary files that the tests generate are now written to 
> /tmp/_giraphTests including zooKeeper files, checkpoints etc. 
> 
> This behavior will be automatically configured whenever 
> InternalVertexRunner.run() or BspCase.prepareJob() is used.
> 
> Usually I can't stop myself once I have my refactoring hat on, therefore I 
> also tidied up a lot of minor stuff, removed code duplications etc.
> 
> 
> This addresses bug GIRAPH-20.
> https://issues.apache.org/jira/browse/GIRAPH-20
> 
> 
> Diffs
> -
> 
>   trunk/src/test/java/org/apache/giraph/TestZooKeeperExt.java 1332106 
>   trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestMutateGraphVertex.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestNotEnoughMapTasks.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestGraphPartitioner.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestAutoCheckpoint.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestBspBasic.java 1332106 
>   trunk/src/test/java/org/apache/giraph/BspCase.java 1332106 
>   trunk/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1332106 
>   trunk/src/main/java/org/apache/giraph/graph/TextAggregatorWriter.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/utils/FileUtils.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/5077/diff
> 
> 
> Testing
> ---
> 
> successfully passed local and pseudo-distributed tests with Hadoop 0.20.203
> 
> 
> Thanks,
> 
> Sebastian
> 
>

Re: Review Request: GIRAPH-20 Move temporary test files from the project directory

2012-05-09 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5077/#review7756
---


Overall, looks great.  Can you address the questions/comments and then I'll 
re-review?


trunk/src/main/java/org/apache/giraph/graph/TextAggregatorWriter.java


Just out of curiosity, why this change?



trunk/src/main/java/org/apache/giraph/utils/FileUtils.java


Why delete it?



trunk/src/test/java/org/apache/giraph/BspCase.java


Empty params and return.



trunk/src/test/java/org/apache/giraph/BspCase.java


Empty params and return.



trunk/src/test/java/org/apache/giraph/BspCase.java


Empty params and return.



trunk/src/test/java/org/apache/giraph/BspCase.java


Empty params and return.



trunk/src/test/java/org/apache/giraph/BspCase.java


@return



trunk/src/test/java/org/apache/giraph/BspCase.java


@return



trunk/src/test/java/org/apache/giraph/TestBspBasic.java


shouldn't it be 49 not 491?


- Avery


On 2012-05-09 11:37:47, Sebastian Schelter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5077/
> ---
> 
> (Updated 2012-05-09 11:37:47)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> All temporary files that the tests generate are now written to 
> /tmp/_giraphTests including zooKeeper files, checkpoints etc. 
> 
> This behavior will be automatically configured whenever 
> InternalVertexRunner.run() or BspCase.prepareJob() is used.
> 
> Usually I can't stop myself once I have my refactoring hat on, therefore I 
> also tidied up a lot of minor stuff, removed code duplications etc.
> 
> 
> This addresses bug GIRAPH-20.
> https://issues.apache.org/jira/browse/GIRAPH-20
> 
> 
> Diffs
> -
> 
>   trunk/src/test/java/org/apache/giraph/TestZooKeeperExt.java 1332106 
>   trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestMutateGraphVertex.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestNotEnoughMapTasks.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestGraphPartitioner.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestAutoCheckpoint.java 1332106 
>   trunk/src/test/java/org/apache/giraph/TestBspBasic.java 1332106 
>   trunk/src/test/java/org/apache/giraph/BspCase.java 1332106 
>   trunk/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1332106 
>   trunk/src/main/java/org/apache/giraph/graph/TextAggregatorWriter.java 
> 1332106 
>   trunk/src/main/java/org/apache/giraph/utils/FileUtils.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/5077/diff
> 
> 
> Testing
> ---
> 
> successfully passed local and pseudo-distributed tests with Hadoop 0.20.203
> 
> 
> Thanks,
> 
> Sebastian
> 
>

Re: Review Request: GIRAPH-20 Move temporary test files from the project directory

2012-05-10 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5077/#review7772
---

Ship it!


+1, thanks Sebastian!  I'm assuming it also passes 'mvn verify', but please 
double check before you commit.

- Avery


On 2012-05-10 09:32:10, Sebastian Schelter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5077/
> ---
> 
> (Updated 2012-05-10 09:32:10)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> All temporary files that the tests generate are now written to 
> /tmp/_giraphTests including zooKeeper files, checkpoints etc. 
> 
> This behavior will be automatically configured whenever 
> InternalVertexRunner.run() or BspCase.prepareJob() is used.
> 
> Usually I can't stop myself once I have my refactoring hat on, therefore I 
> also tidied up a lot of minor stuff, removed code duplications etc.
> 
> 
> This addresses bug GIRAPH-20.
> https://issues.apache.org/jira/browse/GIRAPH-20
> 
> 
> Diffs
> -
> 
>   trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 
> 1336504 
>   trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1336504 
>   trunk/src/main/java/org/apache/giraph/graph/TextAggregatorWriter.java 
> 1336504 
>   trunk/src/main/java/org/apache/giraph/utils/FileUtils.java PRE-CREATION 
>   trunk/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 
> 1336504 
>   trunk/src/test/java/org/apache/giraph/BspCase.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestAutoCheckpoint.java 1336506 
>   trunk/src/test/java/org/apache/giraph/TestBspBasic.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestGraphPartitioner.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java 1336506 
>   trunk/src/test/java/org/apache/giraph/TestMutateGraphVertex.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestNotEnoughMapTasks.java 1336504 
>   trunk/src/test/java/org/apache/giraph/TestZooKeeperExt.java 1336504 
>   trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1336504 
> 
> Diff: https://reviews.apache.org/r/5077/diff
> 
> 
> Testing
> ---
> 
> successfully passed local and pseudo-distributed tests with Hadoop 0.20.203
> 
> 
> Thanks,
> 
> Sebastian
> 
>

Re: Giraph is now an Apache top level project

2012-05-16 Thread Avery Ching


Thanks Owen for helping us through this process!

Avery

On 5/16/12 1:33 PM, Owen O'Malley wrote:

Today the Apache board voted to graduate Giraph to a top level
project. Congratulations, all!

-- Owen

Re: Intention to request a new roller account

2012-05-16 Thread Avery Ching


A blog would be nice.

Avery

On 5/16/12 12:48 PM, Eugene Koontz wrote:

Hi fellow Giraphers,

I've been asked on https://issues.apache.org/jira/browse/INFRA-4377 to
notify you that I'd like to create a new Apache Roller account - (Apache
Roller is the blogging software used by https://blogs.apache.org/).

I think it would be nice to set up a blog for Giraph at some point - I
assume its url would be https://blogs.apache.org/giraph/ .

However, initially and originally, my request doesn't have anything to
do with Giraph: I'm planning on using my account to post an entry
regarding HBase Access Controls to the HBase blog
(http://blogs.apache.org/hbase/). I created this blog post over a year
ago, but the site is now gone. Fortunately it's still archived here:

  
http://web.archive.org/web/20101031022526/http://hbaseblog.com/2010/10/11/secure-hbase-access-controls/
 .

(So, please forgive the somewhat off-topic nature of this mail).

-Eugene

Re: the project wiki works well

2011-08-23 Thread Avery Ching

Thanks for testing it Hyunsik.

Avery

On Aug 23, 2011, at 12:41 AM, Hyunsik Choi wrote:

> The project wiki works well. For test, I copied "Quick Start Guide" in 
> github's giraph into the apache cwiki.
> 
> https://cwiki.apache.org/confluence/display/GIRAPH/Quick+Start+Guide

Re: Will there be JIRA import for Apache Giraph?

2011-08-23 Thread Avery Ching

Hi Henry,

I added the INFRA-3855 as a blocker on the issue.  We definitely don't want to 
have to commit 2x.

Avery

On Aug 23, 2011, at 3:50 PM, Henry Saputra wrote:

> Hi All,
> 
> From Avery comment in one of the JIRA issue, looks like there will be
> existing JIRA import for Giraph?
> 
> If thats the case, we probably should wait until the import happen
> before filing new issues because they will be overwritten by the
> import.
> 
> - Henry

Re: Will there be JIRA import for Apache Giraph?

2011-08-23 Thread Avery Ching

Henry,

Please disregard my answer, I didn't understand your question.  Jakob's answer 
is right, not much to import for issues from GitHub.

Avery

On Aug 23, 2011, at 4:19 PM, Avery Ching wrote:

> Hi Henry,
> 
> I added the INFRA-3855 as a blocker on the issue.  We definitely don't want 
> to have to commit 2x.
> 
> Avery
> 
> On Aug 23, 2011, at 3:50 PM, Henry Saputra wrote:
> 
>> Hi All,
>> 
>> From Avery comment in one of the JIRA issue, looks like there will be
>> existing JIRA import for Giraph?
>> 
>> If thats the case, we probably should wait until the import happen
>> before filing new issues because they will be overwritten by the
>> import.
>> 
>> - Henry
>

Reviewboard for code reviews

2011-08-29 Thread Avery Ching

Anyone know if we have reviewboard access?

Thanks,

Avery

Re: Reviewboard for code reviews

2011-08-29 Thread Avery Ching

https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the

I'll file an INFRA ticket.

Thanks,

Avery

On Aug 29, 2011, at 10:07 PM, Hyunsik Choi wrote:

Looks possible. Some incubator project (e.g., Kafka) already has a
reviewboard group.

Best regards,
--
Hyunsik Choi



On Tue, Aug 30, 2011 at 1:48 PM, Avery Ching 
mailto:ach...@yahoo-inc.com>> wrote:
Anyone know if we have reviewboard access?

Thanks,

Avery

Re: Reviewboard for code reviews

2011-08-30 Thread Avery Ching

Thanks Henry.  I have filed issue

https://issues.apache.org/jira/browse/INFRA-3892

to get reviewboard access.

Avery

On Aug 30, 2011, at 11:35 AM, Henry Saputra wrote:

Hi Avery, yes you should file INFRA ticket to add Giraph as Groups in
reviews board.

I filed tickets to create one for Kafka and Gora.

- Henry

On Mon, Aug 29, 2011 at 10:13 PM, Avery Ching 
mailto:ach...@yahoo-inc.com>> wrote:
https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the

I'll file an INFRA ticket.

Thanks,

Avery

On Aug 29, 2011, at 10:07 PM, Hyunsik Choi wrote:

Looks possible. Some incubator project (e.g., Kafka) already has a
reviewboard group.

Best regards,
--
Hyunsik Choi



On Tue, Aug 30, 2011 at 1:48 PM, Avery Ching 
mailto:ach...@yahoo-inc.com>> wrote:
Anyone know if we have reviewboard access?

Thanks,

Avery

Re: Reviewboard for code reviews

2011-08-30 Thread Avery Ching

Okay, let's make it optional for now.  For me, it definitely helps to visualize 
the changes better.  Also, I think the feedback tool is pretty good.

Avery

On Aug 30, 2011, at 11:52 AM, Henry Saputra wrote:

> Argh I meant "It should just an option to help review and should not
> be required for patches."
> 
> - Henry
> 
> On Tue, Aug 30, 2011 at 11:51 AM, Henry Saputra  
> wrote:
>> +1
>> 
>> It should just optional to help review not required.
>> 
>> - Henry
>> 
>> On Tue, Aug 30, 2011 at 11:48 AM, Jakob Homan  wrote:
>>> We've just gone around on this one for Kafka and, if reviewboard is
>>> provided, it would be good to keep it as an optional part of the
>>> process.  I've had very negative experiences with it, both in Hadoop
>>> and Hive.  If one would like to do a reviewboard review, that's great
>>> - but for those who don't, standard bullet points should suffice.
>>> -jakob
>>> 
>>> 
>>> 
>>> On Tue, Aug 30, 2011 at 11:38 AM, Avery Ching  wrote:
>>>> Thanks Henry.  I have filed issue
>>>> 
>>>> https://issues.apache.org/jira/browse/INFRA-3892
>>>> 
>>>> to get reviewboard access.
>>>> 
>>>> Avery
>>>> 
>>>> On Aug 30, 2011, at 11:35 AM, Henry Saputra wrote:
>>>> 
>>>> Hi Avery, yes you should file INFRA ticket to add Giraph as Groups in
>>>> reviews board.
>>>> 
>>>> I filed tickets to create one for Kafka and Gora.
>>>> 
>>>> - Henry
>>>> 
>>>> On Mon, Aug 29, 2011 at 10:13 PM, Avery Ching 
>>>> mailto:ach...@yahoo-inc.com>> wrote:
>>>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>>>> 
>>>> I'll file an INFRA ticket.
>>>> 
>>>> Thanks,
>>>> 
>>>> Avery
>>>> 
>>>> On Aug 29, 2011, at 10:07 PM, Hyunsik Choi wrote:
>>>> 
>>>> Looks possible. Some incubator project (e.g., Kafka) already has a
>>>> reviewboard group.
>>>> 
>>>> Best regards,
>>>> --
>>>> Hyunsik Choi
>>>> 
>>>> 
>>>> 
>>>> On Tue, Aug 30, 2011 at 1:48 PM, Avery Ching 
>>>> mailto:ach...@yahoo-inc.com>> wrote:
>>>> Anyone know if we have reviewboard access?
>>>> 
>>>> Thanks,
>>>> 
>>>> Avery
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>

Re: dev process

2011-08-31 Thread Avery Ching

I agree that RTC is better for us, although in case of build breakage, I'm okay 
with CTR in that case.

Avery

On Aug 31, 2011, at 9:09 AM, Jakob Homan wrote:

> RTC is definitely the way to go (he said with a weary sigh).
> 
> On Wed, Aug 31, 2011 at 6:32 AM, Owen O'Malley  wrote:
>> All,
>>   It seems that we've implicitly picked review then commit (RTC) instead of 
>> commit then review (CTR). Apache projects allow either approach and I'm fine 
>> with either. We should just state what we are doing.
>>   I'd also like to propose that we keep a CHANGES.txt file that includes who 
>> contributed and committed each patch. I've created GIRAPH-19 to do that.
>> 
>> Thoughts?
>> 
>> -- Owen

Review Request: GIRAPH-27 Mutable static global state in Vertex.java should be refactored

2011-09-08 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1771/
---

Review request for giraph.


Summary
---

Based on Jake's submission 
https://issues.apache.org/jira/secure/attachment/12493654/GIRAPH-27.patch
Couple of small changes:
- Do not expose GraphState to application developers
- Fixing a few formatting issues


This addresses bug GIRAPH-27.
https://issues.apache.org/jira /browse/GIRAPH-27


Diffs
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
 1166925 

Diff: https://reviews.apache.org/r/1771/diff


Testing
---

Unittest and page rank benchmark on Yahoo! grid with 10 workers.


Thanks,

Avery

Re: Review Request: GIRAPH-27 Mutable static global state in Vertex.java should be refactored

2011-09-08 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1771/
---

(Updated 2011-09-09 01:52:18.842526)


Review request for giraph.


Changes
---

Dmitriy correctly pointed out I forgot to add GraphState.java to the diff.


Summary
---

Based on Jake's submission 
https://issues.apache.org/jira/secure/attachment/12493654/GIRAPH-27.patch
Couple of small changes:
- Do not expose GraphState to application developers
- Fixing a few formatting issues


This addresses bug GIRAPH-27.
https://issues.apache.org/jira /browse/GIRAPH-27


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
 1166925 

Diff: https://reviews.apache.org/r/1771/diff


Testing
---

Unittest and page rank benchmark on Yahoo! grid with 10 workers.


Thanks,

Avery

Re: Review Request: GIRAPH-27 Mutable static global state in Vertex.java should be refactored

2011-09-08 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1771/
---

(Updated 2011-09-09 04:44:32.531014)


Review request for giraph.


Changes
---

Make BasicVertex and MutableVertex abstract classes.  BasicVertex has package 
private methods for get/setGraphState().


Summary
---

Based on Jake's submission 
https://issues.apache.org/jira/secure/attachment/12493654/GIRAPH-27.patch
Couple of small changes:
- Do not expose GraphState to application developers
- Fixing a few formatting issues


This addresses bug GIRAPH-27.
https://issues.apache.org/jira /browse/GIRAPH-27


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
 1166925 

Diff: https://reviews.apache.org/r/1771/diff


Testing
---

Unittest and page rank benchmark on Yahoo! grid with 10 workers.


Thanks,

Avery

Re: Review Request: GIRAPH-27 Mutable static global state in Vertex.java should be refactored

2011-09-08 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1771/
---

(Updated 2011-09-09 06:26:15.766329)


Review request for giraph.


Changes
---

Moved as much of the getGraphState() related method implementations from Vertex 
to BasicVertex and MutableVertex.  Other changes for primitive implementations 
can be done in another JIRA.


Summary
---

Based on Jake's submission 
https://issues.apache.org/jira/secure/attachment/12493654/GIRAPH-27.patch
Couple of small changes:
- Do not expose GraphState to application developers
- Fixing a few formatting issues


This addresses bug GIRAPH-27.
https://issues.apache.org/jira /browse/GIRAPH-27


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java
 1166925 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
 1166925 

Diff: https://reviews.apache.org/r/1771/diff


Testing
---

Unittest and page rank benchmark on Yahoo! grid with 10 workers.


Thanks,

Avery

Re: Incubator report is due

2011-09-12 Thread Avery Ching

Sounds good to me.  I'll reach out to Arun and see if he can fill out out the 
ICLA.

Avery

On Sep 12, 2011, at 8:44 AM, Owen O'Malley wrote:

> I'd propose:
> 
> Giraph
> 
> Giraph is a large-scale, fault-tolerant, Bulk Synchronous Parallel
> (BSP)-based graph
> processing framework that runs on Hadoop. Giraph entered the incubator
> in August 2011.
> 
> Project developments:
> 
> * Project website created.
> * Confluence wiki created.
> * Accounts were created for two of the committers.
> * Project is entirely on Apache infrastructure.
> 
> Next steps:
> * Adding new committers.
> * Making a release.
> * One of the initial committers still hasn't filed an ICLA. We either
> need him to move forward or remove him.

Re: Port to YARN: GIRAPH and HAMA

2011-09-13 Thread Avery Ching

Hi Vinod,

Edward and I have chatted about this at times. It sounds better in
theory (both BSP based and adding support for MRv2) than in practice I
think (underlying implementations are quite different). Actually, I
also believe that in the future, Giraph is not going to solely be
BSP-based graph computing. We are also thinking about other underlying
computing models (i.e. streaming (asynchronous) graph processing - see

http://mail-archives.apache.org/mod_mbox/incubator-giraph-user/201109.mbox/%3CCAEVHzWC8b-7RiBjkDiQKjiu-rVBz9=ogeoajxhbclcr5n3+...@mail.gmail.com%3E

But I think today, the issues are the following:

1) Giraph runs completely as a MapReduce job on Hadoop today. This
needs to be maintained to support our current users, who will not likely
move to MRv2 for at least a year.
2) The internals of Giraph are implemented differently than Hama and
would take some time to port to.
3) If we have various graph processing computing models (BSP based,
streams or asynchronous, or a combination), then being on Hama brings
little value for Giraph.

Perhaps more practically, I wonder if it would be possible for someone
from the Hama team to refactor our code a bit to support Hama-style BSP
in Giraph? Certainly would be a pretty cool project...

Avery

On 9/13/11 4:49 AM, Edward J. Yoon wrote:

Quite a while ago, I implemented a clone of Google Pregel simply using
BSPLib[1] and decided to focus on BSP computing engine.

Hama and Giraph projects are differ in slogan but not in kind.

If we made some collaboration, Giraph should be implemented on top of
Hama BSP computing engine.

Otherwise, we will back to square one again.

1. http://markmail.org/thread/4czcgtjupjvpqcqi

On Sun, Sep 11, 2011 at 11:22 PM, Vinod Kumar Vavilapalli
wrote:

Crosspost to hama-dev and giraph-dev.

It was only in my morning time that I was looking at HAMA-431, the port of
Hama to YARN. And one of the tweets reminded me of JIRA issue GIRAPH-13
which is about porting Giraph to YARN.

I was also looking at the Girpah proposal for entry into Apache Incubator.
There is an interesting section there:
{quote}
Relationships with Other Apache Products

Giraph has some overlapping functionality with Apache Hama. However, there
are some significant differences. Giraph focuses on graph-based bulk
synchronous parallel (BSP) computing, while Apache Hama is more for general
purposed BSP computing. Giraph runs on the Hadoop infrastructure, while
Apache Hama uses its own computing framework.
{quote}

I agree with the point about Hama being a general purposed BSP and Giraph
being completely graph oriented. But the later one about the infrastructure
is going to be moot with both Giraph and Hama trying to be ported over to
YARN.

So here's my billion dollar question: Is it possible to implement Girpah's
graph based APIs over the Hama's bsp APIs which both run over a single
Apache BSP implementation over YARN?

I also do see the email thread regarding Hama and Giraph's future
collaboration when Hadoop NextGen aka YARN comes in:
http://s.apache.org/HamaVsGiraph. So are we ready for this yet?

Disclaimer: I come from the Hadoop world, have no idea of Giraph's APIs or
internals except that I see a bsp package in Giraph's source tree. I do know
a tiny bit about Hama's APIs and internal but my expertise is only two days.

Thanks,
+Vinod
(An elephant maintainer trying to see if a Giraffe can be made to ride over
a hippopotamus riding over an elephant)

Re: Port to YARN: GIRAPH and HAMA

2011-09-13 Thread Avery Ching

Maybe it's possible, hard to say what will happen in a year.  However, 
at the same time, porting an application from any of the projects to the 
another should be shouldn't be too difficult since the Pregel API is 
relatively simple.  However, as I mentioned in my original post, I 
imagine that Giraph will support non-BSP graph computing models as well 
in the future (less portable).


Avery

On 9/13/11 12:51 PM, Dan Brickley wrote:

On 13 September 2011 21:43, Dmitriy Ryaboy  wrote:

Dan,
Given how fast we are currently iterating on the API in Giraph, I think
agreeing on a common API across 3 projects is a bit premature at this stage,
unfortunately..

Current velocity aside, ... could such an interface be plausible? e.g.
this time next year?

Dan

Re: Port to YARN: GIRAPH and HAMA

2011-09-14 Thread Avery Ching


Vinod, thanks for your comments.  I've replied inline.

Avery

On 9/14/11 11:09 AM, Vinod Kumar Vavilapalli wrote:

Avery,

Some replies inline to the issues you outlined.


1)  Giraph runs completely as a MapReduce job on Hadoop today.  This needs

to be maintained to support our current users, who will not likely move to
MRv2 for at least a year.
I think what you need is to support Giraph's graph API for your users, but
no, not the underlying implementation. (Or are you leaking MapReduce APIs to
your users?) Sure, you are restricted to the under implementation(Hadoop
MRV1 or MRV2 whenever it gets used) at any point of time, but what we are
discussing is _that_ future when the underlying implementation itself also
moves to MRV2.
I think the takeaway should be that our clients (at Yahoo! and 
elsewhere) are currently using Giraph on MRv1.  While the Giraph API is 
not exposing the underlying infrastructure APIs (i.e. MRv1 and MRv2), we 
still need to support the MRv1 implementation even while we 
begin/complete the port to MRv2.  I imagine that we will need to support 
both MRv1 and MRv2 for a fairly long period of time as the transition to 
MRv2 for a company (i.e. Yahoo!) could take a very long time (i.e. 
anywhere between 8 months to multiple years).  Some of our internal 
clusters at Yahoo! today are still running 0.20.1 for example.

2)  The internals of Giraph are implemented differently than Hama..

Sure, but only at present. My original question is - given a BSP
implementation on a YARN cluster, can GiraphV2(BSP based) be simply
implemented over that or not. If today, GiraphV1 uses (its own) BSP
implementation over mapreduce APIs on Hadoop MRV1 cluster, I can clearly see
how GiraphV2 can be using (HAMA's) BSP implemented over YARN APIs.

In theory this is true.  However, as mentioned previously, we still have 
users on MRv1 and will need to support it for a long time (i.e. at least 
a year, probably more).   Also I'm fairly certain that during the next 
year, we will have non-BSP based graph processing computing models in 
place as well.  For these reasons, it may not make sense to try to put 
Giraph on top of HAMA even when we are both on MRv2.  It's hard to say 
now as it is early.  Let's visit this at a later time.



3)  If we have various graph processing computing models (BSP based,

streams or asynchronous, or a combination), then being on Hama brings little
value for Giraph.
That future isn't there yet. In any case, I'd bet when you get there, lot of
what you have now also wouldn't be an out-of-the-box fit.

 From my perspective (a third person POV), this is what I can conclude.
Giraph's velocity on Hadoop MapReduce may be real the impedence for thinking
about a possible sharing of the bsp based implementation with HAMAV2. Sure,
Giraph has other ideas regarding the computation model itself, but that is a
future that isn't here yet.

I just hope the same velocity isn't an impedance for thinking about the
next-gen version on top of YARN :) The way I see it, porting Giraph to YARN
is also a revolution in itself; most, if not all, of the implementation will
change yet with the API level compatibility. I am still eagerly looking
forward to the port of Giraph to YARN. May be more digging into Giraph
internals may help my cause too.
Giraph does appear to be moving with a fast velocity currently, but we 
have a clear intention to run on top of MRv2.  Please see 
https://issues.apache.org/jira/browse/GIRAPH-13.  Obviously, the MRv2 
changes are much better suited for Giraph and we look forward to the day 
when nearly all Hadoop instances are running MRv2.

If nothing, this discussion atleast helped sharing of some of the ideas
between the two communities.

Thanks all for putting down in your thoughts.
+Vinod


On Wed, Sep 14, 2011 at 11:46 AM, Thomas Jungblut<
thomas.jungb...@googlemail.com>  wrote:


  We are also thinking about other underlying computing models (i.e.

streaming (asynchronous) graph processing - see


That is a really cool idea. But I don't think we are going to focus solely
on graph computing. We want to enable an interface which can be used for it
(straight forward as described in the Pregel Paper), but I think you are
really graph experts- so we don't want to compete with each other :D
Our asynchronous processing (in my opinion) will just enable the sending of
messages within the computation phase. So the BarrierSync is just a little
transition to make sure every task is ready and every message has been send.
Your Vertex locking is a graph-only feature, this won't be effecting us
anyways.


Giraph runs completely as a MapReduce job on Hadoop today.
Allright.

I think our result is the following:
We (Apache Hama) are focussing on the YARN implementation of the BSP
paradigm.
If you want to run Giraph on a real BSP engine later, feel free to put your
stuff on top of that.
As far as I have seen, there is a 100% backward compatibility of YARN, so
your current solution will run on YA

Re: Vertex serialization

2011-09-15 Thread Avery Ching

It should probably be ints instead of longs for now.  I think I was 
thinking of day when it might be possible to have more than 
Integer.MAX_VALUE edges or messages.  But that would break this current 
implementation anyway.


Avery

On 9/15/11 4:49 AM, Claudio Martella wrote:

Hello,

I've noticed that Vertex's Writable implementation (readFields and
write) write longs for map (edges and messages) sizes (which are
ints). Is it for some compatibility reason?

Review Request: GIRAPH-34 Failure of Vertex reflection for putVertexList from GIRAPH-27

2011-09-16 Thread Avery Ching

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1932/
---

Review request for giraph.

Summary
---

The problem shows up when messages are sent locally, and therefore not
reflected. Hence, after the message has been delivered to the other
local vertex, in the next superstep, if the vertex who sent the
message modifies the object, it will modify the message before the
receiving vertex has a chance to access it. Also, combiners maybe
operate on the (shared) message simultaneously with the compute()
method, therefore potential issues could also occur with messages
eventually sent to remote workers. The solution I am proposing is
that all messages should be copied to prevent this error from occuring
as it is pretty tough to debug. An alternative would just be to tell
users to create copies themselves, but it's less convenient and not
intuitive in my opinion.

This addresses bug GIRAPH-34.
https://issues.apache.org/jira /browse/GIRAPH-34

Diffs
-

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java
1171389

Diff: https://reviews.apache.org/r/1932/diff

Testing
---

I passed local unittests (mvn package) and unittests on my local
machine's Hadop instance (mvn package
-Dprop.mapred.job.tracker=localhost:50300).

Thanks,

Avery

Re: Review Request: GIRAPH-34 Failure of Vertex reflection for putVertexList from GIRAPH-27

2011-09-16 Thread Avery Ching

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1932/
---

(Updated 2011-09-16 20:09:21.003725)

Review request for giraph.

Changes
---

Instead of copying the user messages, simply rely on Javadoc to inform users of
the contract with sent message values.

Passed the failing unittest with local Hadoop instance and local unittests.

Summary
---

This addresses bug GIRAPH-34.
https://issues.apache.org/jira /browse/GIRAPH-34

Diffs (updated)
-

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
1171389

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java
1171389

Diff: https://reviews.apache.org/r/1932/diff

Testing
---

I passed local unittests (mvn package) and unittests on my local
machine's Hadop instance (mvn package
-Dprop.mapred.job.tracker=localhost:50300).

Thanks,

Avery

Re: Giraph-13: Porting Giraph to YARN

2011-09-18 Thread Avery Ching


Hi Vinod,

Thank you for your thoughts.  It would be great if your comments were 
put on GIRAPH-13 so they aren't lost.  You and Jakob should sync up to 
see how to proceed on this.


Avery

On 9/18/11 7:37 AM, Vinod Kumar Vavilapalli wrote:

Hi all,

I finished an excursion into Giraph's code and now I kinda know what it
takes to port Giraph over to run on top of YARN.

When  the base Hadoop clusters are replaced by YARN clusters, Giraph will
have two options:
  - *Giraph still works over mapreduce APIs*: Even after moving to YARN
clusters, Giraph can still run over MapreduceV2+YARN. Without any code
changes at all.
  - *Giraph works natively onYARN*: This can be done in such a way that in
the medium term, Giraph can continue to work on both a Hadoop Mapreduce
cluster as well as a YARN cluster. Two visible effects when this effort goes
underway, that I can think of:
 -- There will be some moving around of classes/interface to separate
APIs from implementation details and a bit of reorganisation of code to help
support both GiraphV1 and GiraphV2.
 -- The other thing the port will probably affect is a fork in the
community's attention (depending on how much of the community's eyeballs the
new world grabs as opposed to the stabilization/feature work on GiraphV1).

Now here's the thing. Avery indicated on the other thread(about Giraph over
HAMA) that most of the users of Giraph need to work on top of a hadoop
mapreduce cluster for quite some time. Which I completely agree with, being
a long time maintainer/supporting-dev of Hadoop clusters myself.

Given that concern, before embarking on the port, I thought I'd get opinions
from the community.

Thanks,
+Vinod

Re: Unit tests on real hadoop cluster

2011-09-29 Thread Avery Ching

Actually, to be fair, I've only executed the distributed unittests on my 
own local Hadoop instance.


I just ran the Hadoop unittests against trunk on my local machine to check

mvn test -Dprop.mapred.job.tracker=localhost:50300


Results :

Tests run: 27, Failures: 0, Errors: 0, Skipped: 0

[INFO] 


[INFO] BUILD SUCCESS
[INFO] 


[INFO] Total time: 12:19.143s
[INFO] Finished at: Thu Sep 29 21:55:55 PDT 2011
[INFO] Final Memory: 6M/81M
[INFO] 



Everything should be fine.

Avery

On 9/29/11 5:18 PM, Hyunsik Choi wrote:

I would like to execute unittest on real hadoop cluster.

I try to execute the following command against giraph trunk version.

mvn test -Dprop.mapred.job.tracker=xxx.korea.ac.kr:8021
-Dprop.zookeeper.list=xxx.korea.ac.kr:2181

However, the unit tests are failed as follows:
https://gist.github.com/1252309

I think It may be my fault because the source code is trunk version.

Any suggestion to this will be helpful.

--
Hyunsik Choi
Database Lab, Korea University

Re: [jira] [Commented] (GIRAPH-10) Aggregators are not exported

2011-09-30 Thread Avery Ching


Claudio,

For the AggregatorWriter, shouldn't your case be addressed?  Since only 
a single thread will use the AggregatorWriter, it can dump the 
aggregators to HDFS at every superstep.  Am I missing something?


Avery

On 9/30/11 2:20 AM, Claudio Martella wrote:

The first option sounds like a good idea to me. It fits quite directly
with the current interface.

This design though doesn't fit with my current need for dumping data
to disk. I'll write an email about it with a couple of new questions.

I think that with some help I can produce some code on this JIRA.

On Thu, Sep 29, 2011 at 9:49 PM, Avery Ching (Commented) (JIRA)
  wrote:

[ 
https://issues.apache.org/jira/browse/GIRAPH-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117569#comment-13117569
 ]

Avery Ching commented on GIRAPH-10:
---

This is definitely an open question here.  Here's a quick thought off the top 
of my head.

How about something like:

public interface AggregatorsWriter {
void initialize(TaskAttemptContext context, int superstep) throws 
IOException;

void writeAggregators(Collection>  
aggregators)
throws IOException, InterruptedException;

void close(TaskAttemptContext context)
throws IOException, InterruptedException;
}

Then we could define a method that lets the user select the frequency of how 
frequently to write the aggregators maybe with an enum (ALL_SUPERSTEPS, 
LAST_SUPERSTEP...).

We could easily implement a default AggregatorsWriter that simply dumps the 
name and aggregator values with the registered aggregator name (maybe class 
name too) and then the value.toString().  And users can implement something 
better if they like.

Another alternative would be to support writing aggregators separately by the 
type, but might be a little excessive and could be done laterhopefully 
someone else also chimes in with some thoughts.


Aggregators are not exported


 Key: GIRAPH-10
 URL: https://issues.apache.org/jira/browse/GIRAPH-10
 Project: Giraph
  Issue Type: New Feature
    Reporter: Avery Ching
Priority: Minor

Currently, aggregator values cannot be saved after a Giraph job.  There should 
be a way to do this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: numFlushThreads is 0 when doing a single worker unittest. Changing the minimum to 1.

2011-10-09 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2322/
---

Review request for giraph.


Summary
---

9 unittests failed due to 

java.lang.IllegalArgumentException
at 
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:589)
at 
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:480)
at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:59)
at 
org.apache.giraph.comm.BasicRPCCommunications.(BasicRPCCommunications.java:375)
at 
org.apache.giraph.comm.RPCCommunications.(RPCCommunications.java:68)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:571)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

The issue is that with only a single worker, the ThreadpoolExecutor fails 
because of having 0 threads as an argument.


This addresses bug GIRAPH-48.
https://issues.apache.org/jira /browse/GIRAPH-48


Diffs
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1180785 

Diff: https://reviews.apache.org/r/2322/diff


Testing
---

Passed unittests.


Thanks,

Avery

Re: October 2011 Incubator board report

2011-10-17 Thread Avery Ching

Thanks Chris, looks good to me.

Avery

On Sun, Oct 16, 2011 at 11:45 AM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hey Guys,
>
> I added a stub Giraph report at:
>
> http://wiki.apache.org/incubator/October2011
>
> Please improve it as you see fit.
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>

Re: Relax RTC on web site commits?

2011-10-26 Thread Avery Ching


+1.

On 10/26/11 5:42 PM, Dmitriy Ryaboy wrote:

+1

On Wed, Oct 26, 2011 at 5:13 PM, Jakob Homan  wrote:


Currently we're doing individual JIRAs for each change to the website,
which is a bunch of ceremony for a routine matter.  In GIRAPH-66, we
discussed relaxing this requirement for website changes.  This is an
approach we've used in other projects and it has worked.  I'm +1.
Thoughts?

Re: [jira] [Commented] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps

2011-10-31 Thread Avery Ching

I'd also like to hear the use case. Currently we don't dump messages in
the vertex output format, but maybe there is a similar case to do so?

Avery

On 10/31/11 11:46 AM, Jake Mannix wrote:

Well I guess that gives us one reason to keep it in the API. What's the
reasoning? Are there
static data sets which make the most sense to have "initial messages"
serialized with the
graph, instead of generating them at start?

I guess if what you're modeling is in some sense a "2nd order"
difference/differential
equation, then knowing the state of the graph is not enough information to
uniquely
describe the evolution, you also need the "first derivative" of it's state
(ie the set of
messages it has at any given time).

-jake

On Mon, Oct 31, 2011 at 11:38 AM, Claudio Martella<
claudio.marte...@gmail.com> wrote:

I actually like the idea of having the messages being inserted at
vertex load. Currently I'm actually fighting with this functionality
missing and was going to open and issue sooner or later.

On Mon, Oct 31, 2011 at 6:19 PM, Jake Mannix (Commented) (JIRA)
wrote:

[

https://issues.apache.org/jira/browse/GIRAPH-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140330#comment-13140330]

Jake Mannix commented on GIRAPH-36:
---

1) BspUtils.createVertex(Configuration conf, GraphState

graphState) requires access to the GraphState for instantiation, currently.
We could avoid it by taking that setGraphState() away from that method and
leaving it in wherever it gets first used (GraphMapper?), but why not be
safe, and always set it right after instantiation, so you know that there's
no other place where someone decides to do BspUtils.createVertex(), but
forgets to then setGraphState() on it.

2) I really don't know whether it makes sense to be able to instantiate

"in-flight" messages with vertices. I just wanted to future-proof the API
a little bit by allowing for the possibility. I'm fine either way.

Ensure that subclassing BasicVertex is possible by user apps

Key: GIRAPH-36
URL: https://issues.apache.org/jira/browse/GIRAPH-36
Project: Giraph
Issue Type: Improvement
Components: graph
Affects Versions: 0.70.0
Reporter: Jake Mannix
Assignee: Jake Mannix
Priority: Blocker
Fix For: 0.70.0

Attachments: GIRAPH-36.diff

Original assumptions in Giraph were that all users would subclass

Vertex (which extended MutableVertex extended BasicVertex). Classes which
wish to have application specific data structures (ie. not a TreeMap>) may need to extend either MutableVertex or BasicVertex.
Unfortunately VertexRange extends ArrayList, and there are other
places where the assumption is that vertex classes are either Vertex, or at
least MutableVertex.

Let's make sure the internal APIs allow for BasicVertex to be the base

class.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA

administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa

For more information on JIRA, see:

http://www.atlassian.com/software/jira

--
Claudio Martella
claudio.marte...@gmail.com

Review Request: GIRAPH-64 Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/
---

Review request for giraph and Jake Mannix.


Summary
---

This is filed on behalf of Jakob Homan.


This addresses bug GIRAPH-64.
https://issues.apache.org/jira/browse/GIRAPH-64


Diffs
-

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1196921 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1196921 

Diff: https://reviews.apache.org/r/2691/diff


Testing
---

Don't know.


Thanks,

Avery

Re: Review Request: GIRAPH-64 Create VertexRunner to make it easier to run users' computations

2011-11-02 Thread Avery Ching

Crap.  I may have forgot to add some files.  Let me check when I get home
in an hour.  Sorry.

Sent from my iPhone

On Nov 2, 2011, at 7:43 PM, Jake Mannix  wrote:

  This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2691/

Does this have the whole diff?  Seems like it's missing stuff.  Only
has GraphMapper.java changes and pom.xml...


- Jake

On November 3rd, 2011, 2:29 a.m., Avery Ching wrote:
  Review request for giraph and Jake Mannix.
By Avery Ching.

*Updated 2011-11-03 02:29:54*
Description

This is filed on behalf of Jakob Homan.

  Testing

Don't know.

  *Bugs: * GIRAPH-64 <https://issues.apache.org/jira/browse/GIRAPH-64>
Diffs

   - http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml
   (1196921)
   -
   
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
   (1196921)

View Diff <https://reviews.apache.org/r/2691/diff/>

Review Request: GIRAPH-47 Export Worker's Context/State to vertices through pre/post/Application/Superstep

2011-11-07 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2746/
---

Review request for giraph.


Summary
---

Claudio's patch for GIRAPH-47.


This addresses bug GIRAPH-47.
https://issues.apache.org/jira/browse/GIRAPH-47


Diffs
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/DefaultWorkerContext.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerContext.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestManualCheckpoint.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestMutateGraphVertex.java
 1198865 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java
 1198865 

Diff: https://reviews.apache.org/r/2746/diff


Testing
---

mvn install


Thanks,

Avery

Re: Review Request: GIRAPH-47 Export Worker's Context/State to vertices through pre/post/Application/Superstep

2011-11-07 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2746/#review3082
---


Claudio, really nice stuff here.  Most of my comments are related to indenting. 
 But otherwise, this is a lot better IMO.  Please take a look at 
CODE_CONVENTIONS and fix accordingly.  While the official policy is 2 space, at 
this time, for the 4 space indented files, please keep to 4 spaces for 
consistency.  We will transition everything over at some point.  New files can 
be 2 space (new convention) if desired.


http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6885>

This doesn't need to be static anymore.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6870>

Indenting should be 8 spaces.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6873>

extra line.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6874>

extra line.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6875>

4 spaces.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6871>

4 spaces indenting.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6872>

4 spaces indenting.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
<https://reviews.apache.org/r/2746/#comment6876>

Align to GiraphJob.WORKER_CONTEXT_CLASS



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
<https://reviews.apache.org/r/2746/#comment6877>

VERTEX_COUNT shouldn't be capitalized.  All caps should be reserved for 
only static values.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
<https://reviews.apache.org/r/2746/#comment6878>

EDGE_COUNT shouldn't be capitalized.  All caps should be reserved for only 
static values.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
<https://reviews.apache.org/r/2746/#comment6887>

These no longer need to be static anymore, could be private variables that 
have public accessor method.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
<https://reviews.apache.org/r/2746/#comment6879>

Might want to add a comment about this example.  I.e.

/**
 * Fully runnable example of how to 
 * emit worker data to HDFS during a graph
 * computation.
 */



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
<https://reviews.apache.org/r/2746/#comment6880>

extra line.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
<https://reviews.apache.org/r/2746/#comment6881>

Awesome, I hated this.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
<https://reviews.apache.org/r/2746/#comment6882>

indenting.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/DefaultWorkerContext.java
<https://reviews.apache.org/r/2746/#comment6883>

extra line.



http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerContext.java
<https://reviews.apache.org/r/2746/#comment6884>

Other javadoc has lines in between comment and params (i.e.
    
* superstep starts.
*
* @throws IllegalAccessException


- Avery


On 2011-11-07 19:09:08, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2746/
> ---
> 
> (Updated 2011-11-07 19:09:08)
> 
&g

Re: Review Request: GIRAPH-47 Export Worker's Context/State to vertices through pre/post/Application/Superstep

2011-11-07 Thread Avery Ching



> On 2011-11-07 19:12:55, Avery Ching wrote:
> > Claudio, really nice stuff here.  Most of my comments are related to 
> > indenting.  But otherwise, this is a lot better IMO.  Please take a look at 
> > CODE_CONVENTIONS and fix accordingly.  While the official policy is 2 
> > space, at this time, for the 4 space indented files, please keep to 4 
> > spaces for consistency.  We will transition everything over at some point.  
> > New files can be 2 space (new convention) if desired.
> 
> Claudio Martella wrote:
> Ok, still have to understand a bit the code conventions. Trying to stick 
> to them. Maybe an Eclipse format conf file would help? Could you share yours, 
> if you have one?

Mine is all messed up too.  I have to manually fix some things.


> On 2011-11-07 19:12:55, Avery Ching wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java,
> >  line 150
> > <https://reviews.apache.org/r/2746/diff/1/?file=56632#file56632line150>
> >
> > This doesn't need to be static anymore.
> 
> Claudio Martella wrote:
> Can't make it non static. Won't be able to read from tests.

Sorry I wasn't more clear, I was suggesting that we fix this.  But it's not 
really related to this issue.  So don't worry about it.  Please ignore my 
comment.


> On 2011-11-07 19:12:55, Avery Ching wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java,
> >  lines 91-92
> > <https://reviews.apache.org/r/2746/diff/1/?file=56634#file56634line91>
> >
> > These no longer need to be static anymore, could be private variables 
> > that have public accessor method.
> 
> Claudio Martella wrote:
> Not sure we can do this. How will tests get to their values. Can't access 
> those members if not static.

I was suggesting that we fix this, maybe give the user the worker context at 
the end.  Actually not sure it's the right solution and it's not really related 
to this issue.  So don't worry about it.  Please ignore my comment.


- Avery


-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2746/#review3082
---


On 2011-11-07 19:09:08, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2746/
> ---
> 
> (Updated 2011-11-07 19:09:08)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Claudio's patch for GIRAPH-47.
> 
> 
> This addresses bug GIRAPH-47.
> https://issues.apache.org/jira/browse/GIRAPH-47
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
>  1198865 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/ap

Re: FOSDEM

2011-11-07 Thread Avery Ching

I think it would be great if someone could talk about Giraph at FOSDEM.  
Thanks for volunteering, Claudio.  I'm not planning to be there 
unfortunately.


Avery

On 11/7/11 12:33 PM, Claudio Martella wrote:

Hello list,

I was thinking about submitting for a talk at fosdem with topic Pregel
&  Giraph. Am I overlapping with somebody else?


Best,
Claudio

Review Request: GIRAPH-11 : Improve the graph distribution of Giraph

2011-11-09 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2788/
---

Review request for giraph.


Summary
---

Warning: This is a very large change!

Vertex ranges no longer exist.  A generic partitioner handles the
division of vertex ids to partitions.  As a default, there is a
HashPartitioner and a HashRangePartitioner that will use the hashCode
of a Java object to decide which partition to place the vertex.
Developers can write their own algorithm to determine how to change
the partitioning as well as implement the assignment of partitions to
workers.  All vertices loaded from the input split are sent to the
owner of the partition rather than loaded locally.  This eliminates the
constraint that the vertices must be ordered in the input split.

The checkpoint format has been changed to suit the new partition
style.  Checkpoints are now a lot simpler.  The master will assign
partitions and the workers will only load their own partitions from
the checkpoint.

Unfortunately, the vertex range implementation was baked into almost
every aspect of the code (hence the ridiculous size of this diff).
But now it should be flexible to support several different graph
partitioning schemes (i.e. hash-based, hash-ranged-based, and for
special cases, fully ranged-based).

Sorry for the long delay, but this way pretty involved.


This addresses bug GIRAPH-11.
https://issues.apache.org/jira/browse/GIRAPH-11


Diffs
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1196639 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/RPCCommunications.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ServerInterface.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerCommunications.java
 1196639 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexInputFormat.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexReader.java
 1196639 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java
 1196639 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SuperstepBalancer.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SuperstepHashPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/AutoBalancer.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertexRangeBalancer.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
 1196639 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1198972 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
 1199643 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GlobalStats.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1198972 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphState.java
 1199643 
  
http://svn.apache.org/repos/asf/in

Re: Newbie wanting to get involved with Giraph development

2011-11-10 Thread Avery Ching

Claudio gave great advice.  The only thing I'd like to add is that you 
also might want to consider using reviewboard for throwing up patches 
(https://reviews.apache.org/) as it helps to get line-by-line comments 
and suggestions.  Definitely email the mailing lists if you have 
questions and have fun!


Avery

On 11/10/11 9:40 AM, Claudio Martella wrote:

Hi,

I don't know if there's anything such as an official Development
Process, I can share how I usually do when I contribute to ASF
projects.

(1) Considering there's already an open issue (a ticket in the JIRA)
I'd download from SVN the  version to which the issue applies, I'd
write the fix, test it through unittests that come with the project
(and fix them if necessary) and create the patch (svn diff>
ISSUE.diff is the way I do) that I later attach it to the issue.

(2) If there's not open issue yet about a bug or a feature missing, if
it's trivial (such as the one you've chosen) I'd write the patch as in
(1) and open the issue describing the problem by attaching the fix. If
it's not trivial I'd open the issue and wait for some contribution to
the discussion. Learn from the other issues you find on the JIRA.

Make sure to follow CODE_CONVENTIONS file ( ;-) ).

I'm expecting the experts to give more insights about the process.

Hope this helps,
Claudio

On Thu, Nov 10, 2011 at 5:56 PM, Shaunak Kashyap  wrote:

Hi,

I'm a newbie to Giraph and ASF projects in general. I would like to
help with Giraph development and think I've found the perfect JIRA to
start: https://issues.apache.org/jira/browse/GIRAPH-63.

Before I make any code changes, however, I'd like to know more about
the development process of this project. What is a good place to start
learning about this?

Thank you,

Shaunak

--
"Now the hardness of this world slowly grinds your dreams away /
Makin' a fool's joke out of the promises we make" --- Bruce
Springsteen, "Blood Brothers"

Re: better way to update site?

2011-11-11 Thread Avery Ching


+1.  Having the pre-generated files in svn is good enough for me.

Avery

On 11/11/11 1:36 PM, Jakob Homan wrote:

As Avery documented in GIRAPH-36
(https://issues.apache.org/jira/browse/GIRAPH-35?focusedCommentId=13107195&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13107195)
it's pretty painful to update the site currently.  This is due to the
fact that we're keeping the site within the src tree (as dictated by
mvn), so we can't just check it out, as other, ant/forrest projects
do.  The other project I found that does this is hbase, which avoids
the whole mess by not keeping their site (ie the contents of
people.apache.org/www/incubator/giraph) in svn, but rather generating
the site locally and then copying straight to that directory.

I think this may be a better approach since it avoids the huge churn
of rming and re-creating the whole site structure each time.

In this schema, once the site is updated, run mvn site:site to
generate its contents, verify its correctness, then scp it to
people.apache.org and replace the current directory.  (or rsync it and
be done).  We'll still have all the history of the site, etc., just
none of the hassle.

What do people think?

Re: Review Request: GIRAPH-11 : Improve the graph distribution of Giraph

2011-11-13 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2788/
---

(Updated 2011-11-14 06:56:19.251685)


Review request for giraph.


Changes
---

Updated the diff as per Hyunsik's request to build against recent trunk 
changes.  While I was waiting I added some fixed and additions as well.

Upgrade ZooKeeper to 3.3.3 from 3.3.1.

Fixed bug in PseudoRandomVertexInputFormat.java where the edges are not fully 
added (hasEdge is not the right place to look for the edge).

Fixed bug in BasicRPCCommunications when putting to a local inPartitionMap

Added counter for last checkpointed superstep

Master should refresh the progress every 60 seconds while waiting for workers 
to ensure that the job isn't killed

Fixed bugs in vertexCounter, finishedVertexCoutner, edgeCounter, and 
sentMessages counter not resetting every update (just cumultatively being 
added).

Add additional helpful status messages for debugging.

Turned off speculative execution for Giraph (not a good idea).

Added analysis of the partition balancing for debugging


Summary
---

Warning: This is a very large change!

Vertex ranges no longer exist.  A generic partitioner handles the
division of vertex ids to partitions.  As a default, there is a
HashPartitioner and a HashRangePartitioner that will use the hashCode
of a Java object to decide which partition to place the vertex.
Developers can write their own algorithm to determine how to change
the partitioning as well as implement the assignment of partitions to
workers.  All vertices loaded from the input split are sent to the
owner of the partition rather than loaded locally.  This eliminates the
constraint that the vertices must be ordered in the input split.

The checkpoint format has been changed to suit the new partition
style.  Checkpoints are now a lot simpler.  The master will assign
partitions and the workers will only load their own partitions from
the checkpoint.

Unfortunately, the vertex range implementation was baked into almost
every aspect of the code (hence the ridiculous size of this diff).
But now it should be flexible to support several different graph
partitioning schemes (i.e. hash-based, hash-ranged-based, and for
special cases, fully ranged-based).

Sorry for the long delay, but this way pretty involved.


This addresses bug GIRAPH-11.
https://issues.apache.org/jira/browse/GIRAPH-11


Diffs (updated)
-

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/pom.xml 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PseudoRandomVertexInputFormat.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/RPCCommunications.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ServerInterface.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/WorkerCommunications.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexInputFormat.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/GeneratedVertexReader.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SuperstepBalancer.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SuperstepHashPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/AutoBalancer.java
 1201607 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVerte

Re: Review Request: GIRAPH-11 : Improve the graph distribution of Giraph

2011-11-14 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2788/
---

(Updated 2011-11-14 22:24:27.293676)


Review request for giraph.


Changes
---

Made changes based on Jakob's review.


Summary
---

Warning: This is a very large change!

Vertex ranges no longer exist.  A generic partitioner handles the
division of vertex ids to partitions.  As a default, there is a
HashPartitioner and a HashRangePartitioner that will use the hashCode
of a Java object to decide which partition to place the vertex.
Developers can write their own algorithm to determine how to change
the partitioning as well as implement the assignment of partitions to
workers.  All vertices loaded from the input split are sent to the
owner of the partition rather than loaded locally.  This eliminates the
constraint that the vertices must be ordered in the input split.

The checkpoint format has been changed to suit the new partition
style.  Checkpoints are now a lot simpler.  The master will assign
partitions and the workers will only load their own partitions from
the checkpoint.

Unfortunately, the vertex range implementation was baked into almost
every aspect of the code (hence the ridiculous size of this diff).
But now it should be flexible to support several different graph
partitioning schemes (i.e. hash-based, hash-ranged-based, and for
special cases, fully ranged-based).

Sorry for the long delay, but this way pretty involved.


This addresses bug GIRAPH-11.
https://issues.apache.org/jira/browse/GIRAPH-11


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionStats.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionUtils.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionBalancer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionExchange.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionOwner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/Partition.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/MasterGraphPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashWorkerPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashRangeWorkerPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashPartitionerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashRangePartitionerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/GraphPartitionerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashMasterPartitioner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRangeBalancer.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/BasicPartitionOwner.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/StaticBalancer.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexEdgeCount.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexRange.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
 1201630 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GlobalStats.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1201630 
  
http://svn.a

Re: Review Request: GIRAPH-11 : Improve the graph distribution of Giraph

2011-11-14 Thread Avery Ching

 line 76
> > <https://reviews.apache.org/r/2788/diff/2/?file=57811#file57811line76>
> >
> > typo: dependant -> dependent

Changed.


> On 2011-11-14 20:54:28, Jakob Homan wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionBalancer.java,
> >  line 123
> > <https://reviews.apache.org/r/2788/diff/2/?file=57814#file57814line123>
> >
> > rename: value -> totalValue, to be consistent with usage.

Changed.


> On 2011-11-14 20:54:28, Jakob Homan wrote:
> > http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/RangeWorkerPartitioner.java,
> >  line 117
> > <https://reviews.apache.org/r/2788/diff/2/?file=57824#file57824line117>
> >
> > I'm unclear on this.

RangePartitionerFactory unfortunately is abstract, needs implementations of 
various index types.  A developer can use RangeWorkerPartitioner as something 
to help them out for their particular implementation.  This is somewhat 
experimental work, but the idea is that it will allow very very advanced users 
to customize partiitoning based on a range for their particular index type.  I 
am making this class abstract with a big notice on what needs to be done if you 
want to use it.


- Avery


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2788/#review3211
---


On 2011-11-14 22:24:27, Avery Ching wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2788/
> ---
> 
> (Updated 2011-11-14 22:24:27)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Warning: This is a very large change!
> 
> Vertex ranges no longer exist.  A generic partitioner handles the
> division of vertex ids to partitions.  As a default, there is a
> HashPartitioner and a HashRangePartitioner that will use the hashCode
> of a Java object to decide which partition to place the vertex.
> Developers can write their own algorithm to determine how to change
> the partitioning as well as implement the assignment of partitions to
> workers.  All vertices loaded from the input split are sent to the
> owner of the partition rather than loaded locally.  This eliminates the
> constraint that the vertices must be ordered in the input split.
> 
> The checkpoint format has been changed to suit the new partition
> style.  Checkpoints are now a lot simpler.  The master will assign
> partitions and the workers will only load their own partitions from
> the checkpoint.
> 
> Unfortunately, the vertex range implementation was baked into almost
> every aspect of the code (hence the ridiculous size of this diff).
> But now it should be flexible to support several different graph
> partitioning schemes (i.e. hash-based, hash-ranged-based, and for
> special cases, fully ranged-based).
> 
> Sorry for the long delay, but this way pretty involved.
> 
> 
> This addresses bug GIRAPH-11.
> https://issues.apache.org/jira/browse/GIRAPH-11
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionStats.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionUtils.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionBalancer.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionExchange.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/PartitionOwner.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/Partition.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/MasterGraphPartitioner.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashWorkerPartitioner.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashRangeWorkerPartitioner.java
>  PRE-CREATION 
>   
> http://svn.apache.org/r

Re: Did GIRAPH-11 break vertex reactivation?

2011-11-15 Thread Avery Ching


Yes, I think I broke it.  Sorry.  Let me get you a diff to test quickly.

Avery

On 11/15/11 12:42 PM, Sebastian Schelter wrote:

Hi,

I updated to the latest trunk (after the GIRAPH-11 commit) and wanted to
continue to work on GIRAPH-51 where I use a small toy graph to test
SimpleShortestPathVertex.

Unfortunately my code did not work anymore and I guess I tracked it down
to the fact that vertex that voted to halt are not reacted anymore when
new messages arrive.

In SimpleShortestPathVertex every vertex always votes to halt and only
gets reactivated when a shorter path to it has been found. However my
test run always finished after superstep 0.

I don't know too much about Giraph's internals yet, but my guess is that
the number of sent messages is not tracked correctly anymore. Therefore
giraph finishes the algorithm (as all vertices voted to halt) although
there should still be messages in the pipeline.

I think I tracked it down to this behavior:

GraphMapper declares a variable workerSentMessages = 0 and never
increases it. This variable is given to
BspServiceWorker.finishSuperstep() which writes it to zookeeper and uses
it to compute the GlobalStats afterwards, which are used to decide
whether a new superstep has to be scheduled. As it has never been
increased, the algorithm will always stop when all vertices voted to halt.

It would be great if someone could confirm/disprove this speculation and
help me to continue work on GIRAPH-51

--sebastian

Re: Did GIRAPH-11 break vertex reactivation?

2011-11-15 Thread Avery Ching


This should fix it.  It passed local unittests.  Let me know.

Avery

On 11/15/11 1:03 PM, Avery Ching wrote:

Yes, I think I broke it.  Sorry.  Let me get you a diff to test quickly.

Avery

On 11/15/11 12:42 PM, Sebastian Schelter wrote:

Hi,

I updated to the latest trunk (after the GIRAPH-11 commit) and wanted to
continue to work on GIRAPH-51 where I use a small toy graph to test
SimpleShortestPathVertex.

Unfortunately my code did not work anymore and I guess I tracked it down
to the fact that vertex that voted to halt are not reacted anymore when
new messages arrive.

In SimpleShortestPathVertex every vertex always votes to halt and only
gets reactivated when a shorter path to it has been found. However my
test run always finished after superstep 0.

I don't know too much about Giraph's internals yet, but my guess is that
the number of sent messages is not tracked correctly anymore. Therefore
giraph finishes the algorithm (as all vertices voted to halt) although
there should still be messages in the pipeline.

I think I tracked it down to this behavior:

GraphMapper declares a variable workerSentMessages = 0 and never
increases it. This variable is given to
BspServiceWorker.finishSuperstep() which writes it to zookeeper and uses
it to compute the GlobalStats afterwards, which are used to decide
whether a new superstep has to be scheduled. As it has never been
increased, the algorithm will always stop when all vertices voted to 
halt.


It would be great if someone could confirm/disprove this speculation and
help me to continue work on GIRAPH-51

--sebastian




Index: src/main/java/org/apache/giraph/graph/BspServiceWorker.java
===
--- src/main/java/org/apache/giraph/graph/BspServiceWorker.java (revision 
1202424)
+++ src/main/java/org/apache/giraph/graph/BspServiceWorker.java (working copy)
@@ -548,7 +548,7 @@
 workerGraphPartitioner.finalizePartitionStats(
 partitionStatsList, workerPartitionMap);
 
-finishSuperstep(partitionStatsList, 0);
+finishSuperstep(partitionStatsList);
 }
 
 /**
@@ -773,8 +773,7 @@
 }
 
 @Override
-public boolean finishSuperstep(List partitionStatsList,
-   long workersSentMessages) {
+public boolean finishSuperstep(List partitionStatsList) {
 // This barrier blocks until success (or the master signals it to
 // restart).
 //
@@ -785,8 +784,9 @@
 // of this worker
 // 3. Let the master know it is finished.
 // 4. Then it waits for the master to say whether to stop or not.
+long workerSentMessages = 0;
 try {
-commService.flush(getContext());
+workerSentMessages = commService.flush(getContext());
 } catch (IOException e) {
 throw new IllegalStateException(
 "finishSuperstep: flush failed", e);
@@ -807,7 +807,7 @@
 workerFinishedInfoObj.put(JSONOBJ_PARTITION_STATS_KEY,
   Base64.encodeBytes(partitionStatsBytes));
 workerFinishedInfoObj.put(JSONOBJ_NUM_MESSAGES_KEY,
-  workersSentMessages);
+  workerSentMessages);
 } catch (JSONException e) {
 throw new RuntimeException(e);
 }
Index: src/main/java/org/apache/giraph/graph/GraphMapper.java
===
--- src/main/java/org/apache/giraph/graph/GraphMapper.java  (revision 
1202424)
+++ src/main/java/org/apache/giraph/graph/GraphMapper.java  (working copy)
@@ -512,7 +512,6 @@
 
 List partitionStatsList =
 new ArrayList();
-long workerSentMessages = 0;
 do {
 long superstep = serviceWorker.getSuperstep();
 
@@ -556,7 +555,6 @@
 context.progress();
 
 partitionStatsList.clear();
-workerSentMessages = 0;
 for (Partition partition :
 serviceWorker.getPartitionMap().values()) {
 PartitionStats partitionStats =
@@ -593,8 +591,7 @@
  " maxMem=" + Runtime.getRuntime().maxMemory() +
  " freeMem=" + Runtime.getRuntime().freeMemory());
 }
-} while (!serviceWorker.finishSuperstep(partitionStatsList,
-workerSentMessages));
+} while (!serviceWorker.finishSuperstep(partitionStatsList));
 if (LOG.isInfoEnabled()) {
 LOG.info("map: BSP application done " +
  "(global vertices marked done)");
Index: src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
===
--- src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java

Re: Review Request: GIRAPH-89: Remove debugging system.out from LongDoubleFloatDoubleVertex

2011-11-16 Thread Avery Ching


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2859/#review3304
---

Ship it!


- Avery


On 2011-11-16 20:20:09, shaunak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2859/
> ---
> 
> (Updated 2011-11-16 20:20:09)
> 
> 
> Review request for giraph.
> 
> 
> Summary
> ---
> 
> Removing System.out debugging statement.
> 
> 
> This addresses bug GIRAPH-89.
> https://issues.apache.org/jira/browse/GIRAPH-89
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java
>  1202868 
> 
> Diff: https://reviews.apache.org/r/2859/diff
> 
> 
> Testing
> ---
> 
> $ mvn test
> 
> 
> Thanks,
> 
> shaunak
> 
>

Review Request: GIRAPH-91 - Large-memory improvements (Memory reduced vertex implementation, fast failure, added settings)

2011-11-16 Thread Avery Ching

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2868/
---

Review request for giraph.

Summary
---

There general changes should support larger heap sizes (i.e. >20G)

- Added new EdgeListVertex that stores its edges in a compact pair of lists
instead of Vertex's HashMap.

- Added unittests TestEdgeArrayVertex to test EdgeListVertex.

- Augmented PageRankBenchmark to choose between EdgeListArrayVertex or Vertex
(to try it out).

- Added failure cleanup for failed workers to quickly alert the master that
they are dead by deleting its health ephemeral znode. This allows us to set
higher ZooKeeper timeouts to deal with GC pauses and the like. In a quick test
of 3 nodes, I saw failure in 43 seconds instead of 1m 52 sec.

- Added a context.progress() to flushing to not kill jobs with long timeouts
(GC or lots of messages).

This addresses bug GIRAPH-91.
https://issues.apache.org/jira/browse/GIRAPH-91

Diffs
-

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/EdgeListVertex.java
PRE-CREATION

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java
1202898

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java
PRE-CREATION

Diff: https://reviews.apache.org/r/2868/diff

Testing
---

Local unittests, PageRankBenchmark on multiple machines with >20GB heaps.

Thanks,

Avery

Re: Apache Giraph talk @ FOSDEM

2011-11-21 Thread Avery Ching

Thanks for volunteering Claudio!  It's very nice that Apache Giraph was 
mentioned in the invite, even though it's a relatively new open-source 
project.  I'll read your draft and send you feedback privately sometime 
today.  Also, if you need slides, please feel free to use anything I've 
ever posted (http://www.slideshare.net/averyching).  I think you can 
download them natively.  Also, if you need feedback for your slides, let 
me know.


Avery

On 11/21/11 12:43 PM, Claudio Martella wrote:

Hi devs,

FOSDEM has announced a devroom completely dedicated to Graph Processing:

https://lists.fosdem.org/pipermail/fosdem/2011-November/001344.html

I'm going to submit for a talk there. Here's the draft, feedback is welcome :)

Title: "Apache Giraph: distributed graph processing in the cloud."

Abstract: Web and online social graphs have been rapidly growing in
size and scale during the past decade. In 2008, Google estimated that
the number of web pages reached over a trillion. Online social
networking and email sites, including Yahoo!, Google, Microsoft,
Facebook, LinkedIn, and Twitter, have hundreds of millions of users
and are expected to grow much more in the future. Processing these
graphs plays a big role in relevant and personalized information for
users, such as results from a search engine or news in an online
social networking site.

The Apache Giraph (http://incubator.apache.org/giraph) project is a
faul-tolerant in-memory distributed graph processing system which runs
on top of a standard Hadoop cluster and is capable of running any
standard Bulk Synchronous Parallel (BSP) operation over any large
generic data set which can be represented as a graph. Apache Giraph is
a loose implementation of Google Pregel.
Giraph entered the ASF Incubator in July 2011, where it has enlisted
the aid of committers from Yahoo!, Facebook, LinkedIn, and Twitter.

The talk will present why running MapReduce jobs for graph processing
can be a problem,  introducing the reason why Google designed Pregel
at first place. Later, the BSP model will be presented focusing on how
it can be used to implement a distributed graph processing engine.
The last part of the talk will be dedicated to Apache Giraph, with a
description of the programming model (i.e. the API, some typical
examples such as PageRank and Single Source Shortest Path) along with
a technical overview of how the architecture of Giraph works and how
it leverages the Hadoop infrastructure.


Best,
Claudio

Review Request: GIRAPH-100 - Data input sampling and testing improvements

2011-11-29 Thread Avery Ching

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2959/
---

Review request for giraph.

Summary
---

Got rid of ZooKeeper message for node created on the input split reservation.

Adding some features for debugging:
- Taking only a % of the input splits
- Taking a maximum number of vertices in an input split

Added master status update for number of workers have responded.

Workers will output some information about how the % of input splits that have
been completed.

Fixed a bug where a forced flush of cached vertices in the input split was
happening per input split rather than at the end of processing all input
splits. This requires an additional barrier after processing all the input
splits to allow for the final flush of the cached vertices.

Factored out barrierOnWorkerList to reuse the barrier code coordination by the
master.

Factored out markInputSplitPathFinished to make the code a bit cleaner.

Clearing out the transientInMessages and inMessages maps to reduce processing
time.

Changed the default partition count multipler to produce n^2 partitions rather
than 0.5xn^2 for better balancing when the maximum limit is not exceeded.

Changed SimpleCheckpointVertex to throw an Exception instead of System.exit(-1)
for a faster failure (seconds instead of minutes).

Moved SuperstepHashPartitionerFactory to the examples directory. If it is not
there, the test against a real Hadoop instance will fail from
ClassNotFoundException.

This addresses bug GIRAPH-100.
https://issues.apache.org/jira/browse/GIRAPH-100

Diffs
-

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/TextVertexInputFormat.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/partition/HashMasterPartitioner.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/lib/IdWithValueTextOutputFormat.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SuperstepHashPartitionerFactory.java
PRE-CREATION

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
1207804

http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestGraphPartitioner.java
1207804

Diff: https://reviews.apache.org/r/2959/diff

Testing
---

Passed local and Hadoop instance unittests. Ran PageRankBenchmark on a real
Hadoop cluster.

Thanks,

Avery

Re: Review Request: GIRAPH-100 - Data input sampling and testing improvements

2011-12-01 Thread Avery Ching

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2959/
---

(Updated 2011-12-02 02:55:14.025295)

Review request for giraph.

Changes
---

Moved examples/SuperstepHashPartitionerFactory.java to
integration/SuperstepHashPartitionerFactory.java:

Added a few context.progress() to the communication cycle to avoid task
timeouts.