You can change the log level. warning is a warning. btw, it looks like too many unnecessary logs. let's delete logging.
Sent from my iPhone On Jul 14, 2013, at 10:25 PM, Kostas Xirog <[email protected]> wrote: > Thanks for the reply, > Okay, I understand...The hama messenger que warning still persists > though...I deleted the 'hama.messenger.compression,class' property and > re-started hama... > > > > > On Sun, Jul 14, 2013 at 2:58 PM, Edward J. Yoon <[email protected]>wrote: > >> Disabling message compression by deleting >> 'hama.messenger.compression.class' property in hama-default.xml might >> be helpful. >> >> However, the fundamental problems won't fixed by client-side >> configuration. You need to wait next releases. >> >> On Sun, Jul 14, 2013 at 8:24 PM, Kostas Xirog <[email protected]> >> wrote: >>> I installed 0.6.2 and ran my program, and in 0.6.2 it actually uses more >>> memory than it did before...plus the program needs more time to >> execute... >>> >>> What could I be doing wrong?...The input file that hama takes in is only >>> 172.233.579 Bytes ...Any ideas anyone? >>> >>> Also I'm getting this warning in 0.6.2 : " WARN >>> message.MessageTransferQueueFactory: Message queue is configured on >>> deprecated parameter:hama.messenger.queue.class" >>> >>> Does anyone know how I can fix it and if it might be conflicting with my >>> program's execution in some way? >>> >>> Thank you in advance, >>> Kostas X. >>> >>> >>> On Sun, Jul 14, 2013 at 11:51 AM, Edward J. Yoon <[email protected] >>> wrote: >>> >>>> Please use the latest version. >>>> >>>> On Sun, Jul 14, 2013 at 4:28 PM, Kostas Xirog <[email protected]> >>>> wrote: >>>>> Thanks for your reply, >>>>> >>>>> I don't know what I can actually show you that will be of any >>>> help(except >>>>> from my code which is about 1000 lines), but I'll try to give you guys >>>> the >>>>> basic idea. >>>>> Of course I'm using the hama's graph (implementation of Pregel) for >> this. >>>>> >>>>> My program creates a graph with nodes and edges that both have big >> sets >>>> of >>>>> data (such as recordIds and edge values in each record) , as values. >> The >>>>> basic idea is that I'm running a query on this graph in the form of a >>>> path >>>>> (or subgraph), and the program returns the records that contain this >>>> path, >>>>> as well as the values of each of the records that contain this path. >>>>> >>>>> The compute function executes and only the nodes that are part of the >>>> query >>>>> wake up at first, all others halt. As this happens, I collect the >>>> recordIds >>>>> from the node Values and the edge values from the edges, and when the >> end >>>>> nodes have been reached, the program terminates, I collect the result >>>> from >>>>> the end nodes and write it to the result file... >>>>> >>>>> Is there some way I can access a memory mapping or something?... After >>>>> execution with 400.000 records, the log is: >>>>> >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: The total number of >> supersteps: >>>> 48 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: Counters: 12 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: >>>>> org.apache.hama.graph.GraphJobRunner$GraphJobCounter >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: ITERATIONS=42 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: MULTISTEP_PARTITIONING=4 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: INPUT_VERTICES=1001 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: >>>>> org.apache.hama.bsp.JobInProgress$JobCounter >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: SUPERSTEPS=48 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: LAUNCHED_TASKS=6 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: >>>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: SUPERSTEP_SUM=294 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: IO_BYTES_READ=344290795 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=411231 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=1592 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=1001 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: >> TOTAL_MESSAGES_RECEIVED=1580 >>>>>> 13/07/14 10:17:57 INFO bsp.BSPJobClient: TASK_OUTPUT_RECORDS=1001 >>>>> Job 1 Finished in 3559.706 seconds >>>>> >>>>> >>>>> Any ideas? >>>>> Thanks in advance, >>>>> Kostas X. >>>>> >>>>> >>>>> On Sun, Jul 14, 2013 at 10:08 AM, Chia-Hung Lin < >> [email protected] >>>>> wrote: >>>>> >>>>>> Any chance to show how the code, logic, log, etc. is executed? Others >>>>>> might be able to help spot the issue in underlying infrastructure or >>>>>> somewhere else. >>>>>> >>>>>> On 14 July 2013 15:00, Kostas Xirog <[email protected]> wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I'm running my program with 400.000 records as data and the >> execution >>>>>> takes >>>>>>> 50 minutes whereas the execution of the same query on 200.000 >> records >>>>>>> takes 70 seconds. Any idea why that might be? I've been monitoring >> my >>>>>>> system with the 'top' command, and I see that for these 50 minutes >> the >>>>>>> memory usage is 75.5% and the CPU as at 100 almost constantly... >>>>>>> >>>>>>> I'm running hama in local mode on one machine with 8GB of RAM and 8 >>>> CPUs. >>>>>>> Any idea why that might be? Any ideas of how I can fix it? >>>>>>> >>>>>>> Thanks in advance, >>>>>>> Kostas X. >>>> >>>> >>>> >>>> -- >>>> Best Regards, Edward J. Yoon >>>> @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >>
