Re: Unsubscribe

[EUGEO 2015]

some replay

On Monday, September 21, 2015, Olivier Brun 
wrote:

> Unsubscribe
> Le 21 sept. 2015 02:33, "Jiang Xiaodong"  > a écrit :
>
>> unsubscribe
>>
>> --
>> Thanks
>> -Xiaodong
>>
>>

Re: unsubscribe

[EUGEO 2015]

i dont understand what i need to do in order to test this

On Friday, September 18, 2015, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

>
> Kindly Drop a mail to user-unsubscr...@hadoop.apache.org
> 
>
>
> Thanks & Regards
>
>  Brahma Reddy Battula
>
>
>
>
> --
> *From:* Jean-Eric CAZAMEA [jecaza...@yahoo.com
> ]
> *Sent:* Friday, September 18, 2015 1:17 AM
> *To:* User Hadoop
> *Subject:* unsubscribe
>
>
>

Re: unsubscribe

[Tsuyoshi Ozawa]

Please sent an email to user-unsubscr...@hadoop.apache.org to
unsubscribe hadoop-user mailing list instead of sending msg to
user@hadoop.apache.org.

Thanks,
- Tsuyoshi

On Mon, Sep 21, 2015 at 4:48 PM, EUGEO 2015  wrote:
> i dont understand what i need to do in order to test this
>
>
> On Friday, September 18, 2015, Brahma Reddy Battula
>  wrote:
>>
>>
>> Kindly Drop a mail to user-unsubscr...@hadoop.apache.org
>>
>>
>> Thanks & Regards
>>
>>  Brahma Reddy Battula
>>
>>
>>
>>
>> 
>> From: Jean-Eric CAZAMEA [jecaza...@yahoo.com]
>> Sent: Friday, September 18, 2015 1:17 AM
>> To: User Hadoop
>> Subject: unsubscribe
>>
>>
>

Re: Yarn ResourceManager web UI does not show job

[Boyu Zhang]

Thanks a lot for the answer!

If you don't mind help more on this, here is what I am seeing.

- The NameNode/DataNode and ResourceManager/NodeManager were running for 6
months before I discovered that the job history server was not running.
After bringing up the job history server, I saw like 2k+ jobs showing up
from the history server web ui. But then the job history server got
restarted, and I don't see any jobs more than 7 days old showing up in the
history web ui.

- I've disabled the cleaner in the config file.

My question is, is there a way to find/recover the job history files more
than 7 days old? I read that the container logs are stored locally in the
NodeManger user log dir, and there are files (I have not dig through them
yet). I am not sure if the deleted job history files (by history cleaner)
are not easy to recover.

Thanks in advance,
Boyu


On Mon, Sep 21, 2015 at 4:35 PM, Varun Saxena 
wrote:

> MR jobs will write history files to path given by config
> mapreduce.jobhistory.intermediate-done-dir
> History server will then move them to done dir which is given by config m
> apreduce.jobhistory.done-dir.
>
> By default these config values
> are ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
> and ${yarn.app.mapreduce.am.staging-dir}/history/done respectively.
>
> 7 days is also configurable(config being mapreduce.jobhistory.max-age-ms).
> You can set this value according to your cluster.
>
> I hope this answers your question.
>
> Regards,
> Varun Saxena.
>
> On Tue, Sep 22, 2015 at 1:39 AM, Boyu Zhang  wrote:
>
>> Thanks a lot for the clarification!
>>
>> I tried to find the log and history information about finished jobs. But
>> they are not in hfs://xxx/user/myusername/output/_SUCCESS (0B). Can you
>> please give some pointers on where the statistical/job history files are
>> located? The hfs:///history/done only stores history files up to 7 days.
>>
>> Thanks,
>> Boyu
>>
>> On Mon, Sep 21, 2015 at 1:23 PM, Varun Saxena 
>> wrote:
>>
>>> No, you cant show them in RM UI then.
>>>
>>> However if you can start another daemon, you can consider using YARN
>>> Application History/Timeline Server or MR Job History Server(only for MR
>>> jobs)  to see information about completed jobs.
>>> You can look up Hadoop documentation to learn more about them and how to
>>> configure them.
>>>
>>> Just to clarify though, the apps themselves are not lost, as in, the
>>> output is not lost. Its just the information about them which is no longer
>>> present on RM restart.
>>>
>>> Regards,
>>> Varun Saxena.
>>>
>>> On Mon, Sep 21, 2015 at 10:31 PM, Boyu Zhang 
>>> wrote:
>>>
 Thanks for the answer Varun.

 It is the case that yarn.resourcemanager.recovery.enabled is set to be
 false. Is there a way to show the jobs that are submitted before the
 restart? We don't want to lose that data.

 Thanks,
 Boyu


 On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena  wrote:

> Hi Boyu,
>
> RM stores apps in state store if recovery is enabled. Only then they
> will be available on restart.
> Otherwise they are kept in memory and hence lost on restart.
>
> You may not have it enabled. Check config value for below config. By
> default its false.
> yarn.resourcemanager.recovery.enabled
>
> Regards,
> Varun.
>
> On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang 
> wrote:
>
>> Hello Everyone,
>>
>> I have a strange error regarding the ResourceManager web UI (
>> http://xx.xx:8088).
>>
>> Someone before me set up the hadoop + yarn cluster using Pivotal HD,
>> it was running fine. Then today, the resource manager and node manager
>> disappeared, the logs did not record this. I restarted them, they are up
>> and running, but the resource manger web UI does not show any jobs. We 
>> have
>> 700+ jobs in the past, and they were showing before.
>>
>> If I submit MapReduce jobs, the new submitted ones show up. But the
>> disappear again after restart the resource manger and node manager.
>>
>> Can anyone give any hint on where to look?
>>
>> Thanks in advance,
>> Boyu
>>
>>
>

>>>
>>
>

Re: issue about enable UseNUMA flag in hadoop framework

[Grant Overby (groverby)]

UseNUMA isn't supported by all garbage collectors.

CMS is not supported
G1 is supported
Not sure about others

[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]

Grant Overby
Software Engineer
Cisco.com
grove...@cisco.com
Mobile: 865 724 4910






[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you 
print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.

Please click 
here for 
Company Registration Information.





From: Zhudacai >
Reply-To: "user@hadoop.apache.org" 
>
Date: Saturday, September 19, 2015 at 4:42 AM
To: "user@hadoop.apache.org" 
>, 
"gene...@hadoop.apache.org" 
>, 
"common-iss...@hadoop.apache.org" 
>, 
"yarn-iss...@hadoop.apache.org" 
>, 
"mapreduce-iss...@hadoop.apache.org."
 >
Cc: "dongbo (E)" >, Zhanweitao 
>
Subject: issue about enable UseNUMA flag in hadoop framework

Hi,ALL



We get a problem about enable UseNUMA flag for my hadoop framework.

We've tried to specify JVM flags during hadoop daemon's starts,
e.g. export HADOOP_NAMENODE_OPTS="-XX:+UseNUMA -Dcom.sun.management.jmxremote 
$HADOOP_NAMENODE_OPTS",
export HADOOP_SECONDARYNAMENODE_OPTS="-XX:+UseNUMA 
-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS", etc.
But the ratio between local and remote memory access is 2:1, just remains as 
same as before.

Then we find that hadoop MapReduce start child JVM processes to run task in 
containers. So we passes -XX:+UseNUMA to JVMs by set theting configuration 
parameter child.java.opts. But hadoop starts to throw 
ExitCodeExceptionException (exitCode=1), seems that hadoop does not support 
this JVM parameter.

What should we do to enable UseNUMA flag for my hadoop? Or what should we do to 
decrease the local/remote memory access in NUMA framework? Should we just 
change Hadoop script or resorts to source code? And how to do it?

The hadoop version is 2.6.0.

Best Regards.

Dacai

Re: Yarn ResourceManager web UI does not show job

[Varun Saxena]

Hi Boyu,

RM stores apps in state store if recovery is enabled. Only then they will
be available on restart.
Otherwise they are kept in memory and hence lost on restart.

You may not have it enabled. Check config value for below config. By
default its false.
yarn.resourcemanager.recovery.enabled

Regards,
Varun.

On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang  wrote:

> Hello Everyone,
>
> I have a strange error regarding the ResourceManager web UI (
> http://xx.xx:8088).
>
> Someone before me set up the hadoop + yarn cluster using Pivotal HD, it
> was running fine. Then today, the resource manager and node manager
> disappeared, the logs did not record this. I restarted them, they are up
> and running, but the resource manger web UI does not show any jobs. We have
> 700+ jobs in the past, and they were showing before.
>
> If I submit MapReduce jobs, the new submitted ones show up. But the
> disappear again after restart the resource manger and node manager.
>
> Can anyone give any hint on where to look?
>
> Thanks in advance,
> Boyu
>
>

Re: Yarn ResourceManager web UI does not show job

[Boyu Zhang]

Thanks for the answer Varun.

It is the case that yarn.resourcemanager.recovery.enabled is set to be
false. Is there a way to show the jobs that are submitted before the
restart? We don't want to lose that data.

Thanks,
Boyu


On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena 
wrote:

> Hi Boyu,
>
> RM stores apps in state store if recovery is enabled. Only then they will
> be available on restart.
> Otherwise they are kept in memory and hence lost on restart.
>
> You may not have it enabled. Check config value for below config. By
> default its false.
> yarn.resourcemanager.recovery.enabled
>
> Regards,
> Varun.
>
> On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang 
> wrote:
>
>> Hello Everyone,
>>
>> I have a strange error regarding the ResourceManager web UI (
>> http://xx.xx:8088).
>>
>> Someone before me set up the hadoop + yarn cluster using Pivotal HD, it
>> was running fine. Then today, the resource manager and node manager
>> disappeared, the logs did not record this. I restarted them, they are up
>> and running, but the resource manger web UI does not show any jobs. We have
>> 700+ jobs in the past, and they were showing before.
>>
>> If I submit MapReduce jobs, the new submitted ones show up. But the
>> disappear again after restart the resource manger and node manager.
>>
>> Can anyone give any hint on where to look?
>>
>> Thanks in advance,
>> Boyu
>>
>>
>

Re: Yarn ResourceManager web UI does not show job

[Varun Saxena]

No, you cant show them in RM UI then.

However if you can start another daemon, you can consider using YARN
Application History/Timeline Server or MR Job History Server(only for MR
jobs)  to see information about completed jobs.
You can look up Hadoop documentation to learn more about them and how to
configure them.

Just to clarify though, the apps themselves are not lost, as in, the output
is not lost. Its just the information about them which is no longer present
on RM restart.

Regards,
Varun Saxena.

On Mon, Sep 21, 2015 at 10:31 PM, Boyu Zhang  wrote:

> Thanks for the answer Varun.
>
> It is the case that yarn.resourcemanager.recovery.enabled is set to be
> false. Is there a way to show the jobs that are submitted before the
> restart? We don't want to lose that data.
>
> Thanks,
> Boyu
>
>
> On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena 
> wrote:
>
>> Hi Boyu,
>>
>> RM stores apps in state store if recovery is enabled. Only then they will
>> be available on restart.
>> Otherwise they are kept in memory and hence lost on restart.
>>
>> You may not have it enabled. Check config value for below config. By
>> default its false.
>> yarn.resourcemanager.recovery.enabled
>>
>> Regards,
>> Varun.
>>
>> On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang 
>> wrote:
>>
>>> Hello Everyone,
>>>
>>> I have a strange error regarding the ResourceManager web UI (
>>> http://xx.xx:8088).
>>>
>>> Someone before me set up the hadoop + yarn cluster using Pivotal HD, it
>>> was running fine. Then today, the resource manager and node manager
>>> disappeared, the logs did not record this. I restarted them, they are up
>>> and running, but the resource manger web UI does not show any jobs. We have
>>> 700+ jobs in the past, and they were showing before.
>>>
>>> If I submit MapReduce jobs, the new submitted ones show up. But the
>>> disappear again after restart the resource manger and node manager.
>>>
>>> Can anyone give any hint on where to look?
>>>
>>> Thanks in advance,
>>> Boyu
>>>
>>>
>>
>

Yarn ResourceManager web UI does not show job

[Boyu Zhang]

Hello Everyone,

I have a strange error regarding the ResourceManager web UI (
http://xx.xx:8088).

Someone before me set up the hadoop + yarn cluster using Pivotal HD, it was
running fine. Then today, the resource manager and node manager
disappeared, the logs did not record this. I restarted them, they are up
and running, but the resource manger web UI does not show any jobs. We have
700+ jobs in the past, and they were showing before.

If I submit MapReduce jobs, the new submitted ones show up. But the
disappear again after restart the resource manger and node manager.

Can anyone give any hint on where to look?

Thanks in advance,
Boyu

Hetergeneous Hadoop Cluster

[Ashish Kumar9]

Hi :

Has anyone tried a heterogeneous hadoop cluster with management nodes and 
data nodes running on multiple linux distros. and on multiple h/w 
architecture .

If so , what is the performance of such cluster .

Thanks
Ashish

Re: Yarn ResourceManager web UI does not show job

[Boyu Zhang]

Thanks a lot for the clarification!

I tried to find the log and history information about finished jobs. But
they are not in hfs://xxx/user/myusername/output/_SUCCESS (0B). Can you
please give some pointers on where the statistical/job history files are
located? The hfs:///history/done only stores history files up to 7 days.

Thanks,
Boyu

On Mon, Sep 21, 2015 at 1:23 PM, Varun Saxena 
wrote:

> No, you cant show them in RM UI then.
>
> However if you can start another daemon, you can consider using YARN
> Application History/Timeline Server or MR Job History Server(only for MR
> jobs)  to see information about completed jobs.
> You can look up Hadoop documentation to learn more about them and how to
> configure them.
>
> Just to clarify though, the apps themselves are not lost, as in, the
> output is not lost. Its just the information about them which is no longer
> present on RM restart.
>
> Regards,
> Varun Saxena.
>
> On Mon, Sep 21, 2015 at 10:31 PM, Boyu Zhang 
> wrote:
>
>> Thanks for the answer Varun.
>>
>> It is the case that yarn.resourcemanager.recovery.enabled is set to be
>> false. Is there a way to show the jobs that are submitted before the
>> restart? We don't want to lose that data.
>>
>> Thanks,
>> Boyu
>>
>>
>> On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena 
>> wrote:
>>
>>> Hi Boyu,
>>>
>>> RM stores apps in state store if recovery is enabled. Only then they
>>> will be available on restart.
>>> Otherwise they are kept in memory and hence lost on restart.
>>>
>>> You may not have it enabled. Check config value for below config. By
>>> default its false.
>>> yarn.resourcemanager.recovery.enabled
>>>
>>> Regards,
>>> Varun.
>>>
>>> On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang 
>>> wrote:
>>>
 Hello Everyone,

 I have a strange error regarding the ResourceManager web UI (
 http://xx.xx:8088).

 Someone before me set up the hadoop + yarn cluster using Pivotal HD, it
 was running fine. Then today, the resource manager and node manager
 disappeared, the logs did not record this. I restarted them, they are up
 and running, but the resource manger web UI does not show any jobs. We have
 700+ jobs in the past, and they were showing before.

 If I submit MapReduce jobs, the new submitted ones show up. But the
 disappear again after restart the resource manger and node manager.

 Can anyone give any hint on where to look?

 Thanks in advance,
 Boyu


>>>
>>
>

Re: Yarn ResourceManager web UI does not show job

[Varun Saxena]

MR jobs will write history files to path given by config
mapreduce.jobhistory.intermediate-done-dir
History server will then move them to done dir which is given by config m
apreduce.jobhistory.done-dir.

By default these config values
are ${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
and ${yarn.app.mapreduce.am.staging-dir}/history/done respectively.

7 days is also configurable(config being mapreduce.jobhistory.max-age-ms).
You can set this value according to your cluster.

I hope this answers your question.

Regards,
Varun Saxena.

On Tue, Sep 22, 2015 at 1:39 AM, Boyu Zhang  wrote:

> Thanks a lot for the clarification!
>
> I tried to find the log and history information about finished jobs. But
> they are not in hfs://xxx/user/myusername/output/_SUCCESS (0B). Can you
> please give some pointers on where the statistical/job history files are
> located? The hfs:///history/done only stores history files up to 7 days.
>
> Thanks,
> Boyu
>
> On Mon, Sep 21, 2015 at 1:23 PM, Varun Saxena 
> wrote:
>
>> No, you cant show them in RM UI then.
>>
>> However if you can start another daemon, you can consider using YARN
>> Application History/Timeline Server or MR Job History Server(only for MR
>> jobs)  to see information about completed jobs.
>> You can look up Hadoop documentation to learn more about them and how to
>> configure them.
>>
>> Just to clarify though, the apps themselves are not lost, as in, the
>> output is not lost. Its just the information about them which is no longer
>> present on RM restart.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Mon, Sep 21, 2015 at 10:31 PM, Boyu Zhang 
>> wrote:
>>
>>> Thanks for the answer Varun.
>>>
>>> It is the case that yarn.resourcemanager.recovery.enabled is set to be
>>> false. Is there a way to show the jobs that are submitted before the
>>> restart? We don't want to lose that data.
>>>
>>> Thanks,
>>> Boyu
>>>
>>>
>>> On Mon, Sep 21, 2015 at 12:53 PM, Varun Saxena 
>>> wrote:
>>>
 Hi Boyu,

 RM stores apps in state store if recovery is enabled. Only then they
 will be available on restart.
 Otherwise they are kept in memory and hence lost on restart.

 You may not have it enabled. Check config value for below config. By
 default its false.
 yarn.resourcemanager.recovery.enabled

 Regards,
 Varun.

 On Mon, Sep 21, 2015 at 10:01 PM, Boyu Zhang 
 wrote:

> Hello Everyone,
>
> I have a strange error regarding the ResourceManager web UI (
> http://xx.xx:8088).
>
> Someone before me set up the hadoop + yarn cluster using Pivotal HD,
> it was running fine. Then today, the resource manager and node manager
> disappeared, the logs did not record this. I restarted them, they are up
> and running, but the resource manger web UI does not show any jobs. We 
> have
> 700+ jobs in the past, and they were showing before.
>
> If I submit MapReduce jobs, the new submitted ones show up. But the
> disappear again after restart the resource manger and node manager.
>
> Can anyone give any hint on where to look?
>
> Thanks in advance,
> Boyu
>
>

>>>
>>
>

Re: Hetergeneous Hadoop Cluster

[Tushar Kapila]

Would only matter if OS specific communication was being used between
nodes. I assume they do not do that.
If that is true -> It would depend on the network and each nodes config for
the work it is doing. Cluster performance would not suffer just because it
is heterogeneous.

On Mon, Sep 21, 2015 at 8:09 PM, Ashish Kumar9  wrote:

> Hi :
>
> Has anyone tried a heterogeneous hadoop cluster with management nodes and
> data nodes running on multiple linux distros. and on multiple h/w
> architecture .
>
> If so , what is the performance of such cluster .
>
> Thanks
> Ashish




-- 
Regards
Tushar Kapila