[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg / putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated GIRAPH-185: --- Attachment: (was: GIRAPH-185.patch) > Improve concurrency of putMsg / putMsgList > -- > > Key: GIRAPH-185 > URL: https://issues.apache.org/jira/browse/GIRAPH-185 > Project: Giraph > Issue Type: Improvement > Components: graph >Affects Versions: 0.2.0 >Reporter: Bo Wang >Assignee: Bo Wang > Fix For: 0.2.0 > > Attachments: GIRAPH-185.patch, GIRAPH-185.patch, GIRAPH-185.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently in putMsg / putMsgList, a synchronized closure is used to protect > the whole transientInMessages when adding the new message. This lock prevents > other concurrent calls to putMsg/putMsgList and increases the response time. > We should use fine-grain locks to allow high concurrency in message > communication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg / putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated GIRAPH-185: --- Attachment: GIRAPH-185.patch Sorry for replying late. I've been pretty busy last week with midterms and deadlines. I did a few tests on a 24-core 32GB DRAM machine. Quite astonishing, the performance of the new version (ConcurrentHashMap + ConcurrentLinkedQueue) is kind of slower than the original version. I ran the PageRankBenchmark with "-e 100 -s 10 -V 1". Results are as following. (original) #cores superstep.exec.time 12 11580 8 10651 4 12944 2 19334 1 20782 (ConcurrentHashMap + ConcurrentLinkedQueue) #cores superstep.exec.time 12 11639 8 11527 4 13653 2 20354 1 27825 I think the overhead came from the lock/unlock on the get(), especially when reading messages sent in the previous stage where no lock is needed since it's sequential. To verify my assumption, I changed the ConcurrentLinkedQueue back to ArrayList and wrap it with synchronized, then the performance does improve. (ConcurrentHashMap) #cores superstep.exec.time 12 11380 8 11618 4 12834 2 18417 1 22952 In comparison, it's more or less the same as the original version. Then I ran with another set of parameters "-e 1000 -s 10 -V 1000". It seems the fine grain lock does help. (original) #cores superstep.exec.time 12 97101 6 11321 (ConcurrentHashMap) #cores superstep.exec.time 12 92848 (4.4%) 6 10834 (4.3%) I attached the new patch (ConcurrentHashMap version) for the review. Claudio, thanks for offering to help. You may checkout a clean version and apply the patch ($ patch -p0 -i GIRAPH-185.patch) to test it on the cluster. > Improve concurrency of putMsg / putMsgList > -- > > Key: GIRAPH-185 > URL: https://issues.apache.org/jira/browse/GIRAPH-185 > Project: Giraph > Issue Type: Improvement > Components: graph >Affects Versions: 0.2.0 >Reporter: Bo Wang >Assignee: Bo Wang > Fix For: 0.2.0 > > Attachments: GIRAPH-185.patch, GIRAPH-185.patch, GIRAPH-185.patch, > GIRAPH-185.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently in putMsg / putMsgList, a synchronized closure is used to protect > the whole transientInMessages when adding the new message. This lock prevents > other concurrent calls to putMsg/putMsgList and increases the response time. > We should use fine-grain locks to allow high concurrency in message > communication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg / putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated GIRAPH-185: --- Attachment: GIRAPH-185.patch > Improve concurrency of putMsg / putMsgList > -- > > Key: GIRAPH-185 > URL: https://issues.apache.org/jira/browse/GIRAPH-185 > Project: Giraph > Issue Type: Improvement > Components: graph >Affects Versions: 0.2.0 >Reporter: Bo Wang >Assignee: Bo Wang > Fix For: 0.2.0 > > Attachments: GIRAPH-185.patch, GIRAPH-185.patch, GIRAPH-185.patch, > GIRAPH-185.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently in putMsg / putMsgList, a synchronized closure is used to protect > the whole transientInMessages when adding the new message. This lock prevents > other concurrent calls to putMsg/putMsgList and increases the response time. > We should use fine-grain locks to allow high concurrency in message > communication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg / putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated GIRAPH-185: --- Attachment: GIRAPH-185.patch Thanks for keeping up in the communication, Avery. I used putIfAbsent in the new version (thanks Claudio for the suggestion). Here comes the new patch. > Improve concurrency of putMsg / putMsgList > -- > > Key: GIRAPH-185 > URL: https://issues.apache.org/jira/browse/GIRAPH-185 > Project: Giraph > Issue Type: Improvement > Components: graph >Affects Versions: 0.2.0 >Reporter: Bo Wang >Assignee: Bo Wang > Fix For: 0.2.0 > > Attachments: GIRAPH-185.patch, GIRAPH-185.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently in putMsg / putMsgList, a synchronized closure is used to protect > the whole transientInMessages when adding the new message. This lock prevents > other concurrent calls to putMsg/putMsgList and increases the response time. > We should use fine-grain locks to allow high concurrency in message > communication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-185) Improve concurrency of putMsg / putMsgList
[ https://issues.apache.org/jira/browse/GIRAPH-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated GIRAPH-185: --- Attachment: GIRAPH-185.patch Fixed the problem and wait for review. > Improve concurrency of putMsg / putMsgList > -- > > Key: GIRAPH-185 > URL: https://issues.apache.org/jira/browse/GIRAPH-185 > Project: Giraph > Issue Type: Improvement > Components: graph >Affects Versions: 0.2.0 >Reporter: Bo Wang >Assignee: Bo Wang > Fix For: 0.2.0 > > Attachments: GIRAPH-185.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently in putMsg / putMsgList, a synchronized closure is used to protect > the whole transientInMessages when adding the new message. This lock prevents > other concurrent calls to putMsg/putMsgList and increases the response time. > We should use fine-grain locks to allow high concurrency in message > communication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
