Default of hdfs block size is 64 MB but these days many people usually use 128 MB or 256 MB. If we can reduce memory use, we must do.
On Tue, Nov 20, 2012 at 4:25 PM, Thomas Jungblut <[email protected]> wrote: > Try 32m blocks ;-) > We have to recheck if the pre job partitioner works, I missed checking it. > Am 20.11.2012 08:18 schrieb "Edward" <[email protected]>: > >> Okay. It looks like a memory issue. I'll check tomorrow. >> >> Sent from my iPhone >> >> On Nov 20, 2012, at 3:55 PM, Thomas Jungblut <[email protected]> >> wrote: >> >> > Like in mr you should steer this number by the block size. You can't >> > determine the splits correctly without iterating them. >> > Am 20.11.2012 07:42 schrieb "Thomas Jungblut" <[email protected] >> >: >> > >> >> The splits are never deterministic when we are not splitting across >> block >> >> boundaries. >> >> Am 20.11.2012 07:39 schrieb "Edward J. Yoon" <[email protected]>: >> >> >> >>> changed to use just block nums? >> >>> >> >>> Sadly graph examples on 1 ~ 2 GB random graph are not works well. My >> >>> settings are: >> >>> >> >>> HDFS block size: 256 mb >> >>> bsp.tasks.maximum: 3 >> >>> bsp.child.java.opts: -Xmx15524m >> >>> >> >>> On Tue, Nov 20, 2012 at 2:56 AM, Tommaso Teofili >> >>> <[email protected]> wrote: >> >>>> great job, thanks! >> >>>> Tommaso >> >>>> >> >>>> >> >>>> 2012/11/19 Thomas Jungblut <[email protected]> >> >>>> >> >>>>> done >> >>>>> https://issues.apache.org/jira/browse/HAMA-678 >> >>>>> >> >>>>> >> >>>>> 2012/11/18 Edward J. Yoon <[email protected]> >> >>>>> >> >>>>>> We have to fix fileinput splitter. >> >>>>>> >> >>>>>> On Thu, Nov 15, 2012 at 4:46 PM, Thomas Jungblut >> >>>>>> <[email protected]> wrote: >> >>>>>>> +1. >> >>>>>>> >> >>>>>>> 2012/11/15 Tommaso Teofili <[email protected]> >> >>>>>>> >> >>>>>>>> +1 >> >>>>>>>> >> >>>>>>>> Tommaso >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> 2012/11/15 Edward J. Yoon <[email protected]> >> >>>>>>>> >> >>>>>>>>> Hi all, >> >>>>>>>>> >> >>>>>>>>> Here's new RC for Hama 0.6. >> >>>>>>>>> >> >>>>>>>>> Artifacts: >> >>>>>>>>> http://people.apache.org/~edwardyoon/dist/0.6-RC5/ >> >>>>>>>>> >> >>>>>>>>> SVN Tag: >> >>>>>>>>> http://svn.apache.org/repos/asf/hama/tags/0.6-RC5/ >> >>>>>>>>> >> >>>>>>>>> Please vote for this release. >> >>>>>>>>> >> >>>>>>>>> I'm +1. >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> Best Regards, Edward J. Yoon >> >>>>>>>>> @eddieyoon >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Best Regards, Edward J. Yoon >> >>>>>> @eddieyoon >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, Edward J. Yoon >> >>> @eddieyoon >> >> >> -- Best Regards, Edward J. Yoon @eddieyoon
