[jira] [Created] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
Edward J. Yoon created HAMA-845:
---
Summary: The size() of Spilling Queue returns always
numMessagesWritten
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Reporter: Edward J. Yoon
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869359#comment-13869359
]
Edward J. Yoon commented on HAMA-845:
-
I'll commit directly without patch upload.
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869358#comment-13869358
]
Edward J. Yoon commented on HAMA-845:
-
Unit test also should be changed like:
{code}
assertEquals(queue.size(), 1000);
queue.prepareRead();
Text t;
while ((t = queue.poll()) != null) {
assertTrue(msg.equals(t.toString()));
}
assertEquals(queue.size(), 0);
{code}
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward J. Yoon updated HAMA-845:
Affects Version/s: 0.6.3
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward J. Yoon updated HAMA-845:
Component/s: bsp core
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Assigned] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward J. Yoon reassigned HAMA-845:
---
Assignee: Edward J. Yoon
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869369#comment-13869369 ] Edward J. Yoon commented on HAMA-673: - As I mentioned above, Current YARN module was implemented based on 0.23.1. Our hama-yarn is incompatible with hadoop 2.x. So, first of all, we have to change from 0.23.1 to 2.x stable version. Then, see the syncServer = SyncServiceFactory.getSyncServerRunner(jobConf);. Like this, we can design the pluggable messenger service instead of using hadoop RPC. This issue is about it. Anyway, we have to re-implement the yarn module based on hadoop 2.x version, before thinking about this issue. Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: FYI, Comparison and Evaluation of Open Source Implementations of Pregel and Related Systems
Thanks Song Bai and Ed for your replies, looking forward to Song's contributions and HAMA-843/816 to be done. Tommaso p.s.: I think we need a way of continuously benchmarking our trunk (e.g. setup 2+ machines in distributed mode and run tests / benchmarks against them via Jenkins, but I don't know if that's really feasible via ASF Jenkins). 2014/1/13 Edward J. Yoon edwardy...@apache.org Once HAMA-843 is committed, PageRank performance will be dramatically improved. The scalability issue is related with In-Memory VerticesInfo and Queue. DiskVerticesInfo is now available. Disk/Spilling Queue issues will be fixed soon. And also, Graph package's performance can be improved one more time with HAMA-816. On Mon, Jan 13, 2014 at 1:14 AM, Tommaso Teofili tommaso.teof...@gmail.com wrote: by the way: is there anyone aware of what kind of failures were related to PageRank failures highlighted in the mentioned slides (or know who can we ask)? Tommaso 2014/1/10 Edward J. Yoon edwardy...@apache.org Just FYI, https://cs.uwaterloo.ca/~kdaudjee/courses/cs848/slides/proj/F13/JPV.pdf -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
[jira] [Created] (HAMA-846) Renaming of HadoopMessageManager** classes
Edward J. Yoon created HAMA-846: --- Summary: Renaming of HadoopMessageManager** classes Key: HAMA-846 URL: https://issues.apache.org/jira/browse/HAMA-846 Project: Hama Issue Type: Improvement Components: bsp core Affects Versions: 0.6.3 Reporter: Edward J. Yoon Fix For: 0.7.0 Originally we used Hadoop RPC as a default RPC. To support multiple Hadoop versions, we've added our own RPC implementation. But, we still uses the old names for it. To reduce the confusions, let's rename them like: HamaDefaultMessenger** -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869374#comment-13869374
]
Hudson commented on HAMA-845:
-
SUCCESS: Integrated in Hama-trunk #235 (See
[https://builds.apache.org/job/Hama-trunk/235/])
HAMA-845: The size() of Spilling Queue returns always numMessagesWritten
(edwardyoon) (edwardyoon: rev 1557663)
* /hama/trunk/CHANGES.txt
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
*
/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestSpillingQueue.java
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: FYI, Comparison and Evaluation of Open Source Implementations of Pregel and Related Systems
Not very sure, but it seems JUnitBenchmarks can be integrated to Jekins. On 13 January 2014 17:05, Tommaso Teofili tommaso.teof...@gmail.com wrote: Thanks Song Bai and Ed for your replies, looking forward to Song's contributions and HAMA-843/816 to be done. Tommaso p.s.: I think we need a way of continuously benchmarking our trunk (e.g. setup 2+ machines in distributed mode and run tests / benchmarks against them via Jenkins, but I don't know if that's really feasible via ASF Jenkins). 2014/1/13 Edward J. Yoon edwardy...@apache.org Once HAMA-843 is committed, PageRank performance will be dramatically improved. The scalability issue is related with In-Memory VerticesInfo and Queue. DiskVerticesInfo is now available. Disk/Spilling Queue issues will be fixed soon. And also, Graph package's performance can be improved one more time with HAMA-816. On Mon, Jan 13, 2014 at 1:14 AM, Tommaso Teofili tommaso.teof...@gmail.com wrote: by the way: is there anyone aware of what kind of failures were related to PageRank failures highlighted in the mentioned slides (or know who can we ask)? Tommaso 2014/1/10 Edward J. Yoon edwardy...@apache.org Just FYI, https://cs.uwaterloo.ca/~kdaudjee/courses/cs848/slides/proj/F13/JPV.pdf -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
[jira] [Commented] (HAMA-846) Renaming of HadoopMessageManager** classes
[ https://issues.apache.org/jira/browse/HAMA-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869448#comment-13869448 ] Hudson commented on HAMA-846: - SUCCESS: Integrated in Hama-trunk #236 (See [https://builds.apache.org/job/Hama-trunk/236/]) HAMA-846: Renaming of HadoopMessageManager** classes (edwardyoon: rev 1557681) * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHamaMessageManager.java HAMA-846: Renaming of HadoopMessageManager** classes (edwardyoon: rev 1557680) * /hama/trunk/conf/hama-default.xml * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HamaMessageManager.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HamaMessageManagerImpl.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManagerFactory.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHadoopMessageManager.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHamaMessageManager.java Renaming of HadoopMessageManager** classes -- Key: HAMA-846 URL: https://issues.apache.org/jira/browse/HAMA-846 Project: Hama Issue Type: Improvement Components: bsp core Affects Versions: 0.6.3 Reporter: Edward J. Yoon Fix For: 0.7.0 Originally we used Hadoop RPC as a default RPC. To support multiple Hadoop versions, we've added our own RPC implementation. But, we still uses the old names for it. To reduce the confusions, let's rename them like: HamaDefaultMessenger** -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869549#comment-13869549 ] Skater Xu commented on HAMA-673: Thanks Edward for explaination, Is there any one work on this? May be I can have a try to write some code. 1. YARN master protocol integration 2. YARN container manager protocol integration Is there any doc or work flow description doc for HAMA? Regards, Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870135#comment-13870135
]
Hudson commented on HAMA-845:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #147 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/147/])
HAMA-845: The size() of Spilling Queue returns always numMessagesWritten
(edwardyoon) (edwardyoon: rev 1557663)
* /hama/trunk/CHANGES.txt
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
*
/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestSpillingQueue.java
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-842) Add persistent queue option to JobConf
[
https://issues.apache.org/jira/browse/HAMA-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870136#comment-13870136
]
Hudson commented on HAMA-842:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #147 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/147/])
HAMA-842: Add persistent queue option to JobConf (edwardyoon) (edwardyoon: rev
1557644)
* /hama/trunk/CHANGES.txt
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AbstractMessageManager.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageTransferQueueFactory.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/DiskQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MemoryQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MessageQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SingleLockQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMessageQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestPersistQueue.java
*
/hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java
Add persistent queue option to JobConf
--
Key: HAMA-842
URL: https://issues.apache.org/jira/browse/HAMA-842
Project: Hama
Issue Type: New Feature
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
Attachments: HAMA-842_v01.patch, HAMA-842_v02.patch
By default, incoming (local) queue will be reset in every superstep. It means
that unconsumed messages will be lost. If this is persistent queue,
unconsumed messages will be persisted in incoming queue.
{code}
public final void clearOutgoingQueues() {
if (localQueue != null) {
localQueue.close();
}
localQueue = localQueueForNextIteration.getMessageQueue();
localQueue.prepareRead();
localQueueForNextIteration = getSynchronizedReceiverQueue();
notifyInit();
}
{code}
Simply we can add unconsumed messages to localQueueForNextIteration before
switching local queue.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-846) Renaming of HadoopMessageManager** classes
[ https://issues.apache.org/jira/browse/HAMA-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870134#comment-13870134 ] Hudson commented on HAMA-846: - SUCCESS: Integrated in Hama-Nightly-for-Hadoop-2.x #147 (See [https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/147/]) HAMA-846: Renaming of HadoopMessageManager** classes (edwardyoon: rev 1557681) * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHamaMessageManager.java HAMA-846: Renaming of HadoopMessageManager** classes (edwardyoon: rev 1557680) * /hama/trunk/conf/hama-default.xml * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManager.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HadoopMessageManagerImpl.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HamaMessageManager.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/HamaMessageManagerImpl.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageManagerFactory.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHadoopMessageManager.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHamaMessageManager.java Renaming of HadoopMessageManager** classes -- Key: HAMA-846 URL: https://issues.apache.org/jira/browse/HAMA-846 Project: Hama Issue Type: Improvement Components: bsp core Affects Versions: 0.6.3 Reporter: Edward J. Yoon Fix For: 0.7.0 Originally we used Hadoop RPC as a default RPC. To support multiple Hadoop versions, we've added our own RPC implementation. But, we still uses the old names for it. To reduce the confusions, let's rename them like: HamaDefaultMessenger** -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-845) The size() of Spilling Queue returns always numMessagesWritten
[
https://issues.apache.org/jira/browse/HAMA-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870155#comment-13870155
]
Hudson commented on HAMA-845:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-1.x #1150 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-1.x/1150/])
HAMA-845: The size() of Spilling Queue returns always numMessagesWritten
(edwardyoon) (edwardyoon: rev 1557663)
* /hama/trunk/CHANGES.txt
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
*
/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestSpillingQueue.java
The size() of Spilling Queue returns always numMessagesWritten
--
Key: HAMA-845
URL: https://issues.apache.org/jira/browse/HAMA-845
Project: Hama
Issue Type: Bug
Components: bsp core
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
It should be
{code}
public int size() {
return numMessagesWritten - numMessagesRead;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HAMA-842) Add persistent queue option to JobConf
[
https://issues.apache.org/jira/browse/HAMA-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870156#comment-13870156
]
Hudson commented on HAMA-842:
-
SUCCESS: Integrated in Hama-Nightly-for-Hadoop-1.x #1150 (See
[https://builds.apache.org/job/Hama-Nightly-for-Hadoop-1.x/1150/])
HAMA-842: Add persistent queue option to JobConf (edwardyoon) (edwardyoon: rev
1557644)
* /hama/trunk/CHANGES.txt
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJob.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/AbstractMessageManager.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageTransferQueueFactory.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/DiskQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MemoryQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MessageQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SingleLockQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMessageQueue.java
*
/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
* /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestPersistQueue.java
*
/hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java
Add persistent queue option to JobConf
--
Key: HAMA-842
URL: https://issues.apache.org/jira/browse/HAMA-842
Project: Hama
Issue Type: New Feature
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
Attachments: HAMA-842_v01.patch, HAMA-842_v02.patch
By default, incoming (local) queue will be reset in every superstep. It means
that unconsumed messages will be lost. If this is persistent queue,
unconsumed messages will be persisted in incoming queue.
{code}
public final void clearOutgoingQueues() {
if (localQueue != null) {
localQueue.close();
}
localQueue = localQueueForNextIteration.getMessageQueue();
localQueue.prepareRead();
localQueueForNextIteration = getSynchronizedReceiverQueue();
notifyInit();
}
{code}
Simply we can add unconsumed messages to localQueueForNextIteration before
switching local queue.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HAMA-847) Vertex should provide Counters
Edward J. Yoon created HAMA-847: --- Summary: Vertex should provide Counters Key: HAMA-847 URL: https://issues.apache.org/jira/browse/HAMA-847 Project: Hama Issue Type: Improvement Components: graph Affects Versions: 0.6.3 Reporter: Edward J. Yoon Fix For: 0.7.0 Currently the Vertex is not providing counters. we can use just override the peer.getCounter(). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HAMA-843) Message communication overhead between master aggregation and vertex computation supersteps
[
https://issues.apache.org/jira/browse/HAMA-843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward J. Yoon updated HAMA-843:
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've committed this!
Message communication overhead between master aggregation and vertex
computation supersteps
---
Key: HAMA-843
URL: https://issues.apache.org/jira/browse/HAMA-843
Project: Hama
Issue Type: Improvement
Components: graph
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
Attachments: HAMA-843.patch
Within doAggregationUpdates() method, we sends unconsumed messages to next
superstep using send() method. This is huge overhead.
{code}
// in case we need to sync, we need to replay the messages that already
// are added to the queue. This prevents loosing messages when using
// aggregators.
if (firstVertexMessage != null) {
peer.send(peer.getPeerName(), firstVertexMessage);
}
GraphJobMessage msg = null;
while ((msg = peer.getCurrentMessage()) != null) {
peer.send(peer.getPeerName(), msg);
}
{code}
Once HAMA-842 is done, we can get rid of this overhead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon updated HAMA-673: Attachment: yarn-hama-diagram.pptx See attached flow diagram. And, I'll create a issue for you. Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Attachments: yarn-hama-diagram.pptx Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon resolved HAMA-673. - Resolution: Won't Fix I'll create a new ticket for YARN module. Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Attachments: yarn-hama-diagram.pptx Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HAMA-848) Refactor YARN module for hadoop 2.x stable version
Edward J. Yoon created HAMA-848: --- Summary: Refactor YARN module for hadoop 2.x stable version Key: HAMA-848 URL: https://issues.apache.org/jira/browse/HAMA-848 Project: Hama Issue Type: Improvement Reporter: Edward J. Yoon Current yarn-module was implemented based on Hadoop 0.23.1 version. You'll need to use the Hama RPC instead of Hadoop RPC. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870220#comment-13870220 ] Edward J. Yoon commented on HAMA-673: - If you want to work on YARN integration, please try to fix HAMA-848 first. Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Attachments: yarn-hama-diagram.pptx Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-844) Renaming queue implementations consistently
[ https://issues.apache.org/jira/browse/HAMA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870251#comment-13870251 ] Hudson commented on HAMA-844: - SUCCESS: Integrated in Hama-trunk #238 (See [https://builds.apache.org/job/Hama-trunk/238/]) HAMA-844: Renaming queue implementations consistently (edwardyoon: rev 1557899) * /hama/trunk/c++/src/main/native/examples/conf/matrixmultiplication.xml * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/MessageTransferQueueFactory.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/DiskQueueTransfer.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/DiskTransferProtocolQueue.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MemoryQueueTransfer.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MemoryTransferProtocol.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MessageTransferProtocol.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MessageTransferQueue.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMemoryQueue.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMemoryQueueTransfer.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMessageQueue.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMessageTransferProtocol.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueueTransfer.java * /hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueueTransferProtocol.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestPersistQueue.java * /hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestHamaMessageManager.java * /hama/trunk/core/src/test/java/org/apache/hama/pipes/TestPipes.java * /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java Renaming queue implementations consistently --- Key: HAMA-844 URL: https://issues.apache.org/jira/browse/HAMA-844 Project: Hama Issue Type: Bug Components: bsp core Affects Versions: 0.6.3 Reporter: Edward J. Yoon Assignee: Edward J. Yoon Priority: Minor Fix For: 0.7.0 Renaming queue implementations consistently: - DiskTransferProtocolQueue - MemoryTransferProtocol - SortedMessageTransferProtocol - SpillingQueueTransferProtocol -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HAMA-843) Message communication overhead between master aggregation and vertex computation supersteps
[
https://issues.apache.org/jira/browse/HAMA-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870252#comment-13870252
]
Hudson commented on HAMA-843:
-
SUCCESS: Integrated in Hama-trunk #238 (See
[https://builds.apache.org/job/Hama-trunk/238/])
HAMA-843: Message communication overhead between master aggregation and vertex
computation supersteps (edwardyoon: rev 1557891)
* /hama/trunk/CHANGES.txt
* /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java
* /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java
Message communication overhead between master aggregation and vertex
computation supersteps
---
Key: HAMA-843
URL: https://issues.apache.org/jira/browse/HAMA-843
Project: Hama
Issue Type: Improvement
Components: graph
Affects Versions: 0.6.3
Reporter: Edward J. Yoon
Assignee: Edward J. Yoon
Fix For: 0.7.0
Attachments: HAMA-843.patch
Within doAggregationUpdates() method, we sends unconsumed messages to next
superstep using send() method. This is huge overhead.
{code}
// in case we need to sync, we need to replay the messages that already
// are added to the queue. This prevents loosing messages when using
// aggregators.
if (firstVertexMessage != null) {
peer.send(peer.getPeerName(), firstVertexMessage);
}
GraphJobMessage msg = null;
while ((msg = peer.getCurrentMessage()) != null) {
peer.send(peer.getPeerName(), msg);
}
{code}
Once HAMA-842 is done, we can get rid of this overhead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (HAMA-673) Integrate new service layers
[ https://issues.apache.org/jira/browse/HAMA-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869261#comment-13869261 ] Skater Xu edited comment on HAMA-673 at 1/14/14 3:46 AM: - For current hama based on hadoop 1.x, I think it is managing groom servers by master it self. And also handle a lot jobs at the same time. Wht is HAMA YARN integration plan? was (Author: skater): For current hama based on hadoop 1.x, I think it is managing groom servers by master it self. And also handle a lot jobs at the same time. Wht is HAMA plan? Integrate new service layers Key: HAMA-673 URL: https://issues.apache.org/jira/browse/HAMA-673 Project: Hama Issue Type: Sub-task Components: bsp core, build , yarn Reporter: Thomas Jungblut Attachments: yarn-hama-diagram.pptx Integrate the services, messaging/synchronization/scheduler into YARN. The master must startup all the master services. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HAMA-849) No need to use sorted queue for outgoingQueue
Edward J. Yoon created HAMA-849: --- Summary: No need to use sorted queue for outgoingQueue Key: HAMA-849 URL: https://issues.apache.org/jira/browse/HAMA-849 Project: Hama Issue Type: Improvement Components: bsp core, graph Affects Versions: 0.6.3 Reporter: Edward J. Yoon Assignee: Edward J. Yoon Fix For: 0.7.0 Basically message transfer protocol provides senderQueue and receiverQueue separately. Since only incoming queue should be sorted, there's no need to use sorted queue for outgoingQueue. In graphJobRunner case, we can use diskqueue and sortedqueue to reduce the memory usage until sorted spilling queue is implemented. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HAMA-850) Add ByteArrayQueue
Edward J. Yoon created HAMA-850: --- Summary: Add ByteArrayQueue Key: HAMA-850 URL: https://issues.apache.org/jira/browse/HAMA-850 Project: Hama Issue Type: New Feature Components: bsp core Affects Versions: 0.6.3 Reporter: Edward J. Yoon Fix For: 0.7.0 Memory efficient 'ByteArrayQueue' can be a alternative to DiskQueue which is very slow. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HAMA-849) No need to use sorted queue for outgoingQueue
[ https://issues.apache.org/jira/browse/HAMA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon updated HAMA-849: Status: Patch Available (was: Open) No need to use sorted queue for outgoingQueue -- Key: HAMA-849 URL: https://issues.apache.org/jira/browse/HAMA-849 Project: Hama Issue Type: Improvement Components: bsp core, graph Affects Versions: 0.6.3 Reporter: Edward J. Yoon Assignee: Edward J. Yoon Fix For: 0.7.0 Attachments: HAMA-849.patch Basically message transfer protocol provides senderQueue and receiverQueue separately. Since only incoming queue should be sorted, there's no need to use sorted queue for outgoingQueue. In graphJobRunner case, we can use diskqueue and sortedqueue to reduce the memory usage until sorted spilling queue is implemented. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HAMA-849) No need to use sorted queue for outgoingQueue
[ https://issues.apache.org/jira/browse/HAMA-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward J. Yoon updated HAMA-849: Attachment: HAMA-849.patch attach my patch. No need to use sorted queue for outgoingQueue -- Key: HAMA-849 URL: https://issues.apache.org/jira/browse/HAMA-849 Project: Hama Issue Type: Improvement Components: bsp core, graph Affects Versions: 0.6.3 Reporter: Edward J. Yoon Assignee: Edward J. Yoon Fix For: 0.7.0 Attachments: HAMA-849.patch Basically message transfer protocol provides senderQueue and receiverQueue separately. Since only incoming queue should be sorted, there's no need to use sorted queue for outgoingQueue. In graphJobRunner case, we can use diskqueue and sortedqueue to reduce the memory usage until sorted spilling queue is implemented. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
