[
https://issues.apache.org/jira/browse/SPARK-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099438#comment-14099438
]
Ankur Dave commented on SPARK-1987:
-----------------------------------
[~larryxiao] I was thinking of sorting the 3 primitive arrays directly rather
than first putting them into an array of Edge objects. Though each edge will
then be spread across 3 arrays, I think it shouldn't hurt locality too much,
since we already have to access 2 memory locations per edge (the pointer and
the referenced Edge object). Also, it will be more compact and hopefully make
better use of the cache.
> More memory-efficient graph construction
> ----------------------------------------
>
> Key: SPARK-1987
> URL: https://issues.apache.org/jira/browse/SPARK-1987
> Project: Spark
> Issue Type: Improvement
> Components: GraphX
> Reporter: Ankur Dave
> Assignee: Ankur Dave
>
> A graph's edges are usually the largest component of the graph. GraphX
> currently stores edges in parallel primitive arrays, so each edge should only
> take 20 bytes to store (srcId: Long, dstId: Long, attr: Int). However, the
> current implementation in EdgePartitionBuilder uses an array of Edge objects
> as an intermediate representation for sorting, so each edge additionally
> takes about 40 bytes during graph construction (srcId (8) + dstId (8) + attr
> (4) + uncompressed pointer (8) + object overhead (8) + padding (4)). This
> unnecessarily increases GraphX's memory requirements by a factor of 3.
> To save memory, EdgePartitionBuilder should instead use a custom sort routine
> that operates directly on the three parallel arrays.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
