On Wed, Jul 23, 2014 at 6:31 AM, Stack <[email protected]> wrote:
> On Mon, Jul 21, 2014 at 11:11 PM, Li Li <[email protected]> wrote:
>
>> On Tue, Jul 22, 2014 at 1:57 PM, Stack <[email protected]> wrote:
>> > On Mon, Jul 21, 2014 at 10:53 PM, Li Li <[email protected]> wrote:
>> >
>> >> Sorry, I enter tab and it send my unfinished post. See the following
>> >> mail for answers of other questions.
>> >>
>> >> I forget the exception's detail. It throws exception in terminal.
>> >
>> >
>> > What exception is thrown?
>> I forget it. maybe I can retry it with 8 mapper configuration. it
>> seems like out of memory exception
>>
>
>
> Who OOME'd? The map task or hbase?
map task
>
>
>
>> >
>> >
>> >
>> >> The
>> >> default io.sort.mb is 100 and I set it to 500 to speed up reducer.
>> >
>> >
>> > Do you have to have a reducer? If you could skip the shuffle...
>> I have 8 reducers
>>
>
>
> Do you have to reduce?
yes, I have to do reduce task
>
> Would more reducers make your job run faster?
>
>
>
>> >
>> >
>> >
>> >> So
>> >> I set mapred.child.java.opts to 1g
>> >> The datanode/regionserver has 16GB memory but free memory
>> >
>> >
>> > Does the RS use the 16G?
>> the RS use 8G and there are datanode and tasktracker in this machine
>> >
>>
>
>
> How much for DN and TT? They don't need much usually.
>
>
>
>> >
>> >
>> >> for
>> >> map-reduce is about 5gb. So I can't add more mappers
>> >>
>> >>
>> >> How much RAM in these machines?
>> 16GB
>
>
>
> These your machines or EC2? Can you get bigger machines if EC2?
>
> St.Ack
