[ 
https://issues.apache.org/jira/browse/GIRAPH-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102412#comment-13102412
 ] 

Dmitriy V. Ryaboy commented on GIRAPH-28:
-----------------------------------------

There is something that's eating up memory in the LDFDVertex implementation 
that's not the OpenLongFloatHashMap.

I changed my code to specifically test consumption of the edge map memory, and 
removing all the Vertex complexity, by just putting together 4 implementations 
of a most basic interface:

{code}
 public static abstract class Vertex {
    public abstract void addEdge(Edge<LongWritable, FloatWritable> edge);
  }
{code}

The 4 implementations are "Current" (a map of LongWritables to 
Edge<LongWritable, FloatWritable>), "Just Value" (a map of LongWritable to 
FloatWritable), "Primitive Map" (an OpenLongFloatHashMap), and "Primitive" (two 
arrays of longs and floats, which resize and copy the whole array on every edge 
add -- an obviously untenable, but least memory intensive, implementation).

The code is at https://gist.github.com/1210524

The results are much more sensible. Switching to the PrimitiveMap should be 
huge savings; even without that, getting rid of the duplicated LongWritable is 
quite noticeable.

{code}
Current       : 0       144
Just Value    : 0       144
Primitive Map : 0       272
Primitive     : 0       56
Current       : 1       240
Just Value    : 1       216
Primitive Map : 1       272
Primitive     : 1       72
Current       : 10      1104
Just Value    : 10      864
Primitive Map : 10      528
Primitive     : 10      176
Current       : 100     10704
Just Value    : 100     8304
Primitive Map : 100     3728
Primitive     : 100     1256
Current       : 1000    104272
Just Value    : 1000    80272
Primitive Map : 1000    45976
Primitive     : 1000    12056
Current       : 10000   1025616
Just Value    : 10000   785616
Primitive Map : 10000   301192
Primitive     : 10000   120056
{code}


> Introduce new primitive-specific MutableVertex subclasses
> ---------------------------------------------------------
>
>                 Key: GIRAPH-28
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-28
>             Project: Giraph
>          Issue Type: New Feature
>          Components: graph
>    Affects Versions: 0.70.0
>            Reporter: Jake Mannix
>            Assignee: Jake Mannix
>         Attachments: GIRAPH-28.diff
>
>
> As discussed on the list, 
> MutableVertex<LongWritable,DoubleWritable,FloatWritable,DoubleWritable> (for 
> example) could be highly optimized in its memory footprint if the vertex and 
> edge data were held in a form which minimized Java object usage.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to