hadoop cluster container memory limit

[agc studio]

Hi all,

I am running a EMR cluster with 1 master node and 10 core nodes.

When I go to the dashboard of the hadoop cluster, I each container only has
11.25 GB memory available where as the instance that I use for
it(r3.xlarge) has 30.5 GB of memory.

may I ask, how is this possible and why? Also is it possible to fully
utilise these resources.
I am able to change the settings to utilise the 11.25 GB available memory
but I am wondering about the remainder of the 30.5GB that r3.xlarge offers?
--
HEAP=9216
-Dmapred.child.java.opts=-Xmx${HEAP}m \
-Dmapred.job.map.memory.mb=${HEAP} \
-Dyarn.app.mapreduce.am.resource.mb=1024 \
-Dmapred.cluster.map.memory.mb=${HEAP} \
--
Please see the link of the cluster screenshot. http://imgur.com/a/zFvyw

Where does Hadoop get username and group mapping from for linux shell username and group mapping?

[chen dong]

Hi, 

Currently I am working on a project to enhance the security for the Hadoop 
cluster. Eventually I will use Kerberos and Sentry for authentication and 
authorisation. And the username and group mapping will come from AD/LDAP (?), I 
think so. 

But now I am just learning and trying. I have a question and I haven’t figure 
it out is

where the username/group mapping information come from? 

As far as I know there is no username and group name for Hadoop and username 
and group name come from the client wherever from local client machine or 
Kerberos realm. But it is a little bit vague for me and can I get the 
implementation details here? 

Is this information from the machine where HDFS client is located or from the 
linux shell username and group on name node?  Or it depends on the context - 
even related to data node? What if the data nodes and name nodes have different 
users or user-group mapping in the local boxes. 

Regards,

Dong

Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2

[Ravi Prakash]

Hi Dinesh!

This is obviously a very hazardous situation you are in (if your data is
important), so I'd suggest moving carefully. Make as many backups of as
many things you can.

The usual mechanism that Hadoop uses when upgrading is to rename
directories of the old format and keep them around until the admin
finalizes the upgrade. Here is the relevant method :
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L388
. You will probably have to dig into code and see what operations were
performed, where the failure occurred, and figure out how best to fix it.

In fact for your particular upgrade there were quite substantial changes
not just in the Namenode formats, but also the layouts on the datanodes
(which you may have to work on once you recover your namenode) .
https://issues.apache.org/jira/browse/HDFS-6482 . I'm guessing operations
will take a long time and may not work. Why do you need to rollback? We are
on 2.7.2 and its working fine for us.

HTH
Ravi

On Wed, Oct 12, 2016 at 11:01 PM, Dinesh Kumar Prabakaran <
dineshpv...@gmail.com> wrote:

> Hi Guys,
>
> Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not
> finalize* the upgrade. Now I wished to rollback to 2.5.2 version based on
> reference
> 
> .
>
> Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
> shuts down with the following exception,
>
> *org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
> Unexpected version of storage directory ..\Metadata\data\dfs\namenode.
> Reported: -63. Expecting = -57.*
>
> There is already a task regarding this but the status is *Open*.
> https://issues.apache.org/jira/browse/HDFS-9096
>
> Please let me know are there any *work around *to rollback HDFS from
> rolling upgrade without any issues.
>
> Thanks,
> Dinesh Kumar P
>
>
>
>

Fwd: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2

[Dinesh Kumar Prabakaran]

Hi Guys,

Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not finalize*
the upgrade. Now I wished to rollback to 2.5.2 version based on reference

.

Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
shuts down with the following exception,

*org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory ..\Metadata\data\dfs\namenode. Reported: -63.
Expecting = -57.*

There is already a task regarding this but the status is *Open*.
https://issues.apache.org/jira/browse/HDFS-9096

Please let me know are there any *work around *to rollback HDFS from
rolling upgrade without any issues.

Thanks,
Dinesh Kumar P