Hi Rakesh,
It's a bit hard to spot it, but I see "permission denied" in the stack
trace. Perhaps reviewing local file system permissions for Hadoop's
working directories would reveal something. It might not be the case that
something is deleting the file.
Hope this helps,
--Chris
On Dec 8, 201
Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC nodes.
Otherwise you start paging out things you don't usually want paged out in favor
of a larger filesystem cache.
There is also a vm parameter that controls the minimum size of the free chain,
might want to increase that
Yes but even with a MR running, it is only 36GB heap total out of 64GB
ram. This leaves plenty for OS and caching.
The problem seems to be the OS preferring to cache over giving space to the
applications. Once I drop the caches and rerun the MR job again several
times, it runs perfectly fine.
On
Ok. Thanks for the clarification. It's to run an HBase job, so it will
be one node restriction for me.
JM
2012/12/8, Harsh J :
> In case of HBase, the locality is bound to be restricted to one node
> (the node hosting the region asked for). Otherwise, replication
> affects locality (N options).
>
Are you sure that 24 map slots is a good number for this machine?
Remember that you have three services (DN, TT and HRegionServer) with
with a 12 GB for Heap.
Try to use a lower number of map slots (12 for example) and launch your
MR job again.
Can you share your logs in pastebin?
On Sat 08 Dec
Has anyone experienced a TaskTracker/DataNode behaving like the attached
image?
This was during a MR job (which runs often). Note the extremely high
System CPU time. Upon investigating I saw that out of 64GB ram the system
had allocated almost 45GB to cache!
I did a sudo sh -c "sync ; echo 3 >
Hi,
I have posted the same question to the dev list before but I haven'
heard back from anyone so I figured someone here might be able to shed
some light on it. I have recently been reading Hadoop 1.1.0 source
code to better understand the internals and learned a lot from it, so
far. When I was lo
In case of HBase, the locality is bound to be restricted to one node
(the node hosting the region asked for). Otherwise, replication
affects locality (N options).
On Sat, Dec 8, 2012 at 11:27 PM, Jean-Marc Spaggiari
wrote:
> Hi Harsh,
>
> Thanks for your help.
>
> mapred.fairscheduler.locality.de
Hi Harsh,
Thanks for your help.
mapred.fairscheduler.locality.delay seems to be working very well for
me. I have set it with 60s and JoInProgress picked up only "Choosing
data-local task"... It seems to do the job for my usecase. And as you
are saying, if I'm loosing a node while the job is runni
Hello,
I would like to run Hadoop on linux nodes with /etc/hosts like this:
127.0.0.1 localhost
127.0.1.1 hostname.domainname hostname
My administrator says, he needs the second line because of Kerberos. I
tried to LD_PRELOAD a modified version of gettaddrinfo, but it works onl
Answer depends on a couple of features to be present in your version
of Hadoop, and is inline.
On Fri, Dec 7, 2012 at 11:38 PM, Jean-Marc Spaggiari
wrote:
> Hi,
>
> Is there a way for force the tasks from a MR job to run ONLY on the
> taskservers where the input split location is?
There is no se
Hi Tsuyoshi,
For which version of Hadoop is that? I think it's for 0.2x.x, right?
Because I'm not able to find this class in 1.0.x
Thanks,
JM
2012/12/8, Tsuyoshi OZAWA :
> Hi Hioryuki,
>
> Lately I've changed scheduler for improving hadoop, so I may help you.
>
> RMContainerAllocator#handleEven
Hi Hioryuki,
Lately I've changed scheduler for improving hadoop, so I may help you.
RMContainerAllocator#handleEvent decides MapTasks to allocated containers.
You can implement semi-strict(best effort allocation) mode by hacking
there. Note that, however, allocation of containers is done
by Res
13 matches
Mail list logo
