The cost of a superstep described in Wikipedia[1] is as below: The cost of a superstep S = max(Wi) + max(hg) + l = max cost of the local computation + global communication + cost of the barrier synchronization. As you already might know, max(Wi) means just the wait time for the slowest process.
> III - In relation to the cost of computation, is there any way to implement on > Apache Hama? Sure, you can calculate the cost of each superstep using Counters, and can sum finally. With this, you can also add something like "estimated time remaining". Please see our implementation of sync() method[2] (especially PeerCounter.TIME_IN_SYNC_MS). 1. http://en.wikipedia.org/wiki/Bulk_synchronous_parallel 2. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java On Wed, Mar 27, 2013 at 11:26 PM, Júlio Pires <[email protected]> wrote: > Hi Suraj, > > Thanks for the response. > I've been by analyzing and I got the following doubts: > > I - How can I know when a super step ended? > II - With the information of finish of a superstep, it is possible to > capture the costs discussed in the previous email (computing, communication > and synchronization)? > III - In relation to the cost of computation, is there any way to implement on > Apache Hama? > > Thanks! > Júlio > > > 2013/3/16 Suraj Menon <[email protected]> > >> Hi Julio, >> >> - Cost of the longest running local computation; >> We don't have this yet per superstep. >> >> - Cost of global communication between the processors; >> We provide information on count/size in bytes of messages sent and received >> by peers.( IO_BYTES_READ, MESSAGE_BYTES_TRANSFERED, MESSAGE_BYTES_RECEIVED, >> TOTAL_MESSAGES_SENT, TOTAL_MESSAGES_RECEIVED, COMPRESSED_BYTES_SENT, >> COMPRESSED_BYTES_RECEIVED) >> - Cost of the barrier synchronisation at the end of the superstep. >> This is again provided in the final output in TIME_IN_SYNC_MS. >> >> The stats above are aggregated throughout the lifecycle and we don't have a >> breakdown per superstep yet. You may create a JIRA for tracking. >> >> Thanks, >> Suraj >> >> >> On Fri, Mar 15, 2013 at 8:10 PM, Edward J. Yoon <[email protected] >> >wrote: >> >> > Hi, of course. >> > >> > On Fri, Mar 15, 2013 at 9:39 PM, Júlio Pires <[email protected]> >> > wrote: >> > > Hi, >> > > >> > > Thanks for the response. >> > > >> > > So, thinking in a more abstract level, there are means to achieve these >> > > three cost information on Apache Hama (another type of information) >> (see >> > > >> > >> http://en.wikipedia.org/wiki/Bulk_synchronous_parallelThe_Cost_of_a_BSP_algorithm >> > )? >> > > >> > > - Cost of the longest running local computation; >> > > - Cost of global communication between the processors; >> > > - Cost of the barrier synchronisation at the end of the superstep. The >> > cost >> > > of one superstep for p processors >> > > >> > > Thanks! >> > > >> > > >> > > 2013/3/14 Edward J. Yoon <[email protected]> >> > > >> > >> Unfortunately, we currently don't have any monitoring tool for metrics >> > >> such as CPU, memory. >> > >> >> > >> For the cluster and BSP job monitoring, there's a web UI but not >> mature >> > >> yet. >> > >> >> > >> On Fri, Mar 15, 2013 at 2:17 AM, Júlio Pires <[email protected]> >> > >> wrote: >> > >> > Hi, >> > >> > >> > >> > It's possible to retrieve information of performance such as CPU >> > usage or >> > >> > memory in each task on each peer? If so, how do? >> > >> > >> > >> > Also, details of the functioning of job monitoring in Apache Hama. >> > >> > >> > >> > Thanks! >> > >> > Júlio >> > >> >> > >> >> > >> >> > >> -- >> > >> Best Regards, Edward J. Yoon >> > >> @eddieyoon >> > >> >> > >> > >> > >> > -- >> > Best Regards, Edward J. Yoon >> > @eddieyoon >> > >> -- Best Regards, Edward J. Yoon @eddieyoon
