Re: [DISCUSS] Hama 0.7.0
+1 sounds great. Anastasis On 18 Νοε 2013, at 9:01 π.μ., Tommaso Teofili tommaso.teof...@gmail.com wrote: +1 sounds good to me. Tommaso 2013/11/18 Edward J. Yoon edwardy...@apache.org I would like to suggest that we solve the messaging scalability issue. WDYT? The Spilling Queue seems works fine on my cluster. So, if we finish the below tasks, I think, Hama will be very Competitive in terms of the performance and scalability of the both (pure) BSP and Graph computing engine. HAMA-734 Hama Message Manager should be able to delegate the ownership of internal message queue on request for future superstep. HAMA-723 Implement sorting in Spilling queue. HAMA-816 Add the getMsgIterators method for efficient message looping. HAMA-783 Efficient InMemory Storage for Vertices. If no objection, i would like to arrange the JIRA tasks for 0.7.0, based on this. WDYT? On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin cli...@googlemail.com wrote: That looks fine to me. In addition to this, if that task is accomplished, are we planning to release a new version (e.g. a minor version plus 1)? Just to check as it seems that we have demands on frequent releases so that users who need some specific patches can use it earlier. On 3 September 2013 09:45, Edward J. Yoon edwardy...@apache.org wrote: According to Suraj's dependency diagram, asynchronous messaging is most important and highest priority for us at the moment. How about we focus on this one? (Of course, some committers can dedicated on doing GPU, ML algorithms, or Interface Refactoring issues, regardless of *core* roadmap). If we agree with this plan, I think we can separate the async messaging into smaller sub-tasks: * Decision of whether we will use existing open source, or not. * Design the asynchronous messaging interface (maybe (spilling) message queue also should be considered together?). * Implementation of asynchronous messaging functions, such as send or flush. * Evaluation and adopt asynchronous messaging as a default messenger system. WDYT? On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin cli...@googlemail.com wrote: +1 BTW, are we going to prioritize tasks in roadmap? On 28 August 2013 14:17, Tommaso Teofili tommaso.teof...@gmail.com wrote: sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
Re: [DISCUSS] Hama 0.7.0
I would like to suggest that we solve the messaging scalability issue. WDYT? The Spilling Queue seems works fine on my cluster. So, if we finish the below tasks, I think, Hama will be very Competitive in terms of the performance and scalability of the both (pure) BSP and Graph computing engine. HAMA-734 Hama Message Manager should be able to delegate the ownership of internal message queue on request for future superstep. HAMA-723 Implement sorting in Spilling queue. HAMA-816 Add the getMsgIterators method for efficient message looping. HAMA-783 Efficient InMemory Storage for Vertices. If no objection, i would like to arrange the JIRA tasks for 0.7.0, based on this. WDYT? On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin cli...@googlemail.com wrote: That looks fine to me. In addition to this, if that task is accomplished, are we planning to release a new version (e.g. a minor version plus 1)? Just to check as it seems that we have demands on frequent releases so that users who need some specific patches can use it earlier. On 3 September 2013 09:45, Edward J. Yoon edwardy...@apache.org wrote: According to Suraj's dependency diagram, asynchronous messaging is most important and highest priority for us at the moment. How about we focus on this one? (Of course, some committers can dedicated on doing GPU, ML algorithms, or Interface Refactoring issues, regardless of *core* roadmap). If we agree with this plan, I think we can separate the async messaging into smaller sub-tasks: * Decision of whether we will use existing open source, or not. * Design the asynchronous messaging interface (maybe (spilling) message queue also should be considered together?). * Implementation of asynchronous messaging functions, such as send or flush. * Evaluation and adopt asynchronous messaging as a default messenger system. WDYT? On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin cli...@googlemail.com wrote: +1 BTW, are we going to prioritize tasks in roadmap? On 28 August 2013 14:17, Tommaso Teofili tommaso.teof...@gmail.com wrote: sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
Re: [DISCUSS] Hama 0.7.0
+1 sounds good to me. Tommaso 2013/11/18 Edward J. Yoon edwardy...@apache.org I would like to suggest that we solve the messaging scalability issue. WDYT? The Spilling Queue seems works fine on my cluster. So, if we finish the below tasks, I think, Hama will be very Competitive in terms of the performance and scalability of the both (pure) BSP and Graph computing engine. HAMA-734 Hama Message Manager should be able to delegate the ownership of internal message queue on request for future superstep. HAMA-723 Implement sorting in Spilling queue. HAMA-816 Add the getMsgIterators method for efficient message looping. HAMA-783 Efficient InMemory Storage for Vertices. If no objection, i would like to arrange the JIRA tasks for 0.7.0, based on this. WDYT? On Tue, Sep 10, 2013 at 12:11 AM, Chia-Hung Lin cli...@googlemail.com wrote: That looks fine to me. In addition to this, if that task is accomplished, are we planning to release a new version (e.g. a minor version plus 1)? Just to check as it seems that we have demands on frequent releases so that users who need some specific patches can use it earlier. On 3 September 2013 09:45, Edward J. Yoon edwardy...@apache.org wrote: According to Suraj's dependency diagram, asynchronous messaging is most important and highest priority for us at the moment. How about we focus on this one? (Of course, some committers can dedicated on doing GPU, ML algorithms, or Interface Refactoring issues, regardless of *core* roadmap). If we agree with this plan, I think we can separate the async messaging into smaller sub-tasks: * Decision of whether we will use existing open source, or not. * Design the asynchronous messaging interface (maybe (spilling) message queue also should be considered together?). * Implementation of asynchronous messaging functions, such as send or flush. * Evaluation and adopt asynchronous messaging as a default messenger system. WDYT? On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin cli...@googlemail.com wrote: +1 BTW, are we going to prioritize tasks in roadmap? On 28 August 2013 14:17, Tommaso Teofili tommaso.teof...@gmail.com wrote: sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
Re: [DISCUSS] Hama 0.7.0
+1 BTW, are we going to prioritize tasks in roadmap? On 28 August 2013 14:17, Tommaso Teofili tommaso.teof...@gmail.com wrote: sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon
Re: [DISCUSS] Hama 0.7.0
According to Suraj's dependency diagram, asynchronous messaging is most important and highest priority for us at the moment. How about we focus on this one? (Of course, some committers can dedicated on doing GPU, ML algorithms, or Interface Refactoring issues, regardless of *core* roadmap). If we agree with this plan, I think we can separate the async messaging into smaller sub-tasks: * Decision of whether we will use existing open source, or not. * Design the asynchronous messaging interface (maybe (spilling) message queue also should be considered together?). * Implementation of asynchronous messaging functions, such as send or flush. * Evaluation and adopt asynchronous messaging as a default messenger system. WDYT? On Mon, Sep 2, 2013 at 11:31 PM, Chia-Hung Lin cli...@googlemail.com wrote: +1 BTW, are we going to prioritize tasks in roadmap? On 28 August 2013 14:17, Tommaso Teofili tommaso.teof...@gmail.com wrote: sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
Re: [DISCUSS] Hama 0.7.0
sure, it looks reasonable to me. Tommaso 2013/8/28 Edward J. Yoon edwardy...@apache.org Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon
[DISCUSS] Hama 0.7.0
Hi all, After we release the 0.6.3 (HDFS 2.0 version), we have to work for 0.7.0 version now. I would like to suggest that we solve the messaging scalability issue. WDYT? ... And, according to my experiments, BSP framework shows very nice performance (I tested also GraphLab and Spark). Only Graph job is slow. So, I'll mainly work on improving the performance of GraphJobRunner. -- Best Regards, Edward J. Yoon @eddieyoon
