[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773928#comment-13773928 ] Hudson commented on HAMA-732: - SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #40 (See [https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/40/]) HAMA-732 - applied patch for integration with DM for OffHeapVerticesInfo (tommaso: rev 1525198) * /hama/trunk/graph/src/main/java/org/apache/hama/graph/OffHeapVerticesInfo.java * /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestOffHeapVerticesInfo.java HAMA-732 - applied patch for integration with DM for OffHeapVerticesInfo (tommaso: rev 1525197) * /hama/trunk/graph/pom.xml * /hama/trunk/graph/src/main/java/org/apache/hama/graph/DefaultVertexOutputWriter.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java * /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.6.3 > > Attachments: HAMA-732.2.patch, HAMA-732.3.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773748#comment-13773748 ] Hudson commented on HAMA-732: - SUCCESS: Integrated in Hama-trunk #176 (See [https://builds.apache.org/job/Hama-trunk/176/]) HAMA-732 - applied patch for integration with DM for OffHeapVerticesInfo (tommaso: rev 1525198) * /hama/trunk/graph/src/main/java/org/apache/hama/graph/OffHeapVerticesInfo.java * /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestOffHeapVerticesInfo.java HAMA-732 - applied patch for integration with DM for OffHeapVerticesInfo (tommaso: rev 1525197) * /hama/trunk/graph/pom.xml * /hama/trunk/graph/src/main/java/org/apache/hama/graph/DefaultVertexOutputWriter.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java * /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.6.3 > > Attachments: HAMA-732.2.patch, HAMA-732.3.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773741#comment-13773741 ] Tommaso Teofili commented on HAMA-732: -- given last RC has been rollbacked I think we can integrate this for 0.6.3. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.6.3 > > Attachments: HAMA-732.2.patch, HAMA-732.3.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768300#comment-13768300 ] Tommaso Teofili commented on HAMA-732: -- patch updated to use upcoming DM 0.2 release (currently being voted), if no one objects I'd like to commit it to trunk as soon as Hama 0.6.3 is out (and DM release happened of course). > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.2.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13715270#comment-13715270 ] Tommaso Teofili commented on HAMA-732: -- here's a first patch (done with Git, so you may need to use _patch -p1 < file.patch_) which can be applied to trunk as soon as we have a release of DM (currently using _0.2-SNAPSHOT_) > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.2.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713666#comment-13713666 ] Tommaso Teofili commented on HAMA-732: -- an up to date (and passing the tests) of the integration with DM : https://github.com/tteofili/hama/compare/apache:trunk...HAMA-732 > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.2.patch, HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602364#comment-13602364 ] Tommaso Teofili commented on HAMA-732: -- I'll work more on this in the next weeks however the current branch version is not completed, as soon as I've a first draft implementation working I can merge it on trunk. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602097#comment-13602097 ] Edward J. Yoon commented on HAMA-732: - IMO, now you can just merge this to trunk if this can be adjusted to interface with VerticesInfo. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578317#comment-13578317 ] Edward J. Yoon commented on HAMA-732: - Yes, I'll. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578316#comment-13578316 ] Tommaso Teofili commented on HAMA-732: -- I think it'd be worth to abstract the storage of Vertices so that it can be configured plugged on a per BSP basis. In the meantime [~udanax] would it be possible for you to try this patch out on one of your clusters and see if it helps improving graph performance? > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576478#comment-13576478 ] Tommaso Teofili commented on HAMA-732: -- maybe we can have something like custom Hama cache module so that we're not bound to DM / off-heap caching. There're different kind of serializers in DM (see http://svn.apache.org/repos/asf/directmemory/trunk/serializers/) and the number of allocated buffers is defined by configuration of CacheService by using DirectMemory (see http://svn.apache.org/repos/asf/directmemory/trunk/directmemory-cache/src/main/java/org/apache/directmemory/DirectMemory.java) > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573551#comment-13573551 ] Suraj Menon commented on HAMA-732: -- I must admit, I had a very wrong notion about what DirectMemory does. I looked into the code. Once we move our communication to Netty, we can use any serializers we want. Does it have a ByteBuffer pool? We need it to budget/reuse the use of off heap buffers. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573276#comment-13573276 ] Tommaso Teofili commented on HAMA-732: -- that could be an interesting idea, maybe better as hama-offheap in case we choose to use not only DM facilities. I'll try to do some benchmarks and report here. > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13572739#comment-13572739 ] Suraj Menon commented on HAMA-732: -- I like the idea. Can we have a generic storage strategy defined inside a new hama-directmemory module that could be used in other modules say examples and ml? > Investigate the usage of Apache DirectMemory to store vertices > -- > > Key: HAMA-732 > URL: https://issues.apache.org/jira/browse/HAMA-732 > Project: Hama > Issue Type: Improvement > Components: graph >Affects Versions: 0.6.0 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili >Priority: Minor > Fix For: 0.7.0 > > Attachments: HAMA-732.patch > > > Since Apache DirectMemory can keep objects outside of the java heap that > could help on improving performances. > I created a simple extension implementation of VerticesInfo which uses Apache > DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
