Each tasks needed 10 GB heap to handle 256 MB block ... Please review this small patch : https://issues.apache.org/jira/secure/attachment/12554481/mapToList.patch
On Wed, Nov 21, 2012 at 11:09 AM, Edward J. Yoon <[email protected]> wrote: >>> (currentTime - tip.lastPingedTimestamp) > monitorPeriod) > > And, above condition is so sensitive. > > http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/GroomServer.java > > On Wed, Nov 21, 2012 at 9:54 AM, Edward J. Yoon <[email protected]> wrote: >> Solved by set HAMA_HEAPSIZE to 4 GB. >> >> On Tue, Nov 20, 2012 at 8:37 PM, Edward J. Yoon <[email protected]> >> wrote: >>> Default of hdfs block size is 64 MB but these days many people usually >>> use 128 MB or 256 MB. If we can reduce memory use, we must do. >>> >>> On Tue, Nov 20, 2012 at 4:25 PM, Thomas Jungblut >>> <[email protected]> wrote: >>>> Try 32m blocks ;-) >>>> We have to recheck if the pre job partitioner works, I missed checking it. >>>> Am 20.11.2012 08:18 schrieb "Edward" <[email protected]>: >>>> >>>>> Okay. It looks like a memory issue. I'll check tomorrow. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On Nov 20, 2012, at 3:55 PM, Thomas Jungblut <[email protected]> >>>>> wrote: >>>>> >>>>> > Like in mr you should steer this number by the block size. You can't >>>>> > determine the splits correctly without iterating them. >>>>> > Am 20.11.2012 07:42 schrieb "Thomas Jungblut" <[email protected] >>>>> >: >>>>> > >>>>> >> The splits are never deterministic when we are not splitting across >>>>> block >>>>> >> boundaries. >>>>> >> Am 20.11.2012 07:39 schrieb "Edward J. Yoon" <[email protected]>: >>>>> >> >>>>> >>> changed to use just block nums? >>>>> >>> >>>>> >>> Sadly graph examples on 1 ~ 2 GB random graph are not works well. My >>>>> >>> settings are: >>>>> >>> >>>>> >>> HDFS block size: 256 mb >>>>> >>> bsp.tasks.maximum: 3 >>>>> >>> bsp.child.java.opts: -Xmx15524m >>>>> >>> >>>>> >>> On Tue, Nov 20, 2012 at 2:56 AM, Tommaso Teofili >>>>> >>> <[email protected]> wrote: >>>>> >>>> great job, thanks! >>>>> >>>> Tommaso >>>>> >>>> >>>>> >>>> >>>>> >>>> 2012/11/19 Thomas Jungblut <[email protected]> >>>>> >>>> >>>>> >>>>> done >>>>> >>>>> https://issues.apache.org/jira/browse/HAMA-678 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> 2012/11/18 Edward J. Yoon <[email protected]> >>>>> >>>>> >>>>> >>>>>> We have to fix fileinput splitter. >>>>> >>>>>> >>>>> >>>>>> On Thu, Nov 15, 2012 at 4:46 PM, Thomas Jungblut >>>>> >>>>>> <[email protected]> wrote: >>>>> >>>>>>> +1. >>>>> >>>>>>> >>>>> >>>>>>> 2012/11/15 Tommaso Teofili <[email protected]> >>>>> >>>>>>> >>>>> >>>>>>>> +1 >>>>> >>>>>>>> >>>>> >>>>>>>> Tommaso >>>>> >>>>>>>> >>>>> >>>>>>>> >>>>> >>>>>>>> 2012/11/15 Edward J. Yoon <[email protected]> >>>>> >>>>>>>> >>>>> >>>>>>>>> Hi all, >>>>> >>>>>>>>> >>>>> >>>>>>>>> Here's new RC for Hama 0.6. >>>>> >>>>>>>>> >>>>> >>>>>>>>> Artifacts: >>>>> >>>>>>>>> http://people.apache.org/~edwardyoon/dist/0.6-RC5/ >>>>> >>>>>>>>> >>>>> >>>>>>>>> SVN Tag: >>>>> >>>>>>>>> http://svn.apache.org/repos/asf/hama/tags/0.6-RC5/ >>>>> >>>>>>>>> >>>>> >>>>>>>>> Please vote for this release. >>>>> >>>>>>>>> >>>>> >>>>>>>>> I'm +1. >>>>> >>>>>>>>> >>>>> >>>>>>>>> -- >>>>> >>>>>>>>> Best Regards, Edward J. Yoon >>>>> >>>>>>>>> @eddieyoon >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> -- >>>>> >>>>>> Best Regards, Edward J. Yoon >>>>> >>>>>> @eddieyoon >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> -- >>>>> >>> Best Regards, Edward J. Yoon >>>>> >>> @eddieyoon >>>>> >> >>>>> >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
