Query : Hadoop Cluster OS upgrade

[Brahma Reddy Battula]

Hi All,


Does anybody tried out/share learnings ,using maintenance state or  upgrade
domains for big data cluster OS upgrades?



Regards,
Brahma

Re: unsubscribe

[Brahma Reddy Battula]

Please drop mail to "user-unsubscr...@hadoop.apache.org" as mentioned in
the footer mail.

On Fri, Feb 9, 2024 at 2:32 PM Henning Blohm 
wrote:

> unsubscribe
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: user-h...@hadoop.apache.org
>
>

Re: Performance with large no of files

[Brahma Reddy Battula]

Not sure, what's your backup approach.  One option can be archiving[1] the
files which were done for yarn logs[2].
To Speed on this, you can write one mapreduce job for archiving the files.
Please refer to the Document for sample mapreduce[3].


1.https://hadoop.apache.org/docs/stable/hadoop-archives/HadoopArchives.html
2.
https://hadoop.apache.org/docs/stable/hadoop-archive-logs/HadoopArchiveLogs.html
3.
https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

On Sun, Oct 9, 2022 at 9:22 AM Ayush Saxena  wrote:

> Using DistCp is the only option AFAIK. Distcp does support webhdfs, then
> try playing with the number of mappers and so to tune it for better
> performance
>
> -Ayush
>
>
> On 09-Oct-2022, at 8:56 AM, Abhishek  wrote:
>
> 
> Hi,
> We want to backup large no of hadoop small files (~1mn) with webhdfs API
> We are getting a performance bottleneck here and it's taking days to back
> it up.
> Anyone know any solution where performance could be improved using any xml
> settings?
> This would really help us.
> v 3.1.1
>
> Appreciate your help !!
>
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
> ~
> *Abhishek...*
>
>

Re: IPV6 support in Apache Hadoop

[Brahma Reddy Battula]

Hi,

Thanks for reaching out.
There was a development which was done following the umbrella. It might
take some time to merge the trunk as there are some outstanding jira's.
https://issues.apache.org/jira/browse/HADOOP-11890

On Mon, May 2, 2022 at 10:47 AM Deepti Sharma S
 wrote:

> Hello Team,
>
>
>
> Does Apache Hadoop is currently supported on IPV6 network, if yes in which
> version it have support?
>
> We found below links which states that It have support on IPV4 only.
>
>
>
>
> https://cwiki.apache.org/confluence/display/HADOOP2/HadoopIPv6#:~:text=Apache%20Hadoop%20is%20not%20currently,only%2C%20you%20will%20encounter%20problems
> .
>
> https://issues.apache.org/jira/browse/HADOOP-11890
>
>
>
>
>
>
>
> Regards,
>
> Deepti Sharma
> * PMP® & ITIL*
>
>
>


-- 



--Brahma Reddy Battula

Re: Any comment on the log4j issue?

[Brahma Reddy Battula]

CVE-2021-44228 states that, it will affect the Apache Log4j2 2.0-beta9
through 2.12.1 and 2.13.0 through 2.15.0 JNDI features used in
configuration, log messages, and parameters do not protect against attacker
controlled LDAP and other JNDI related endpoints *And hadoop uses the
log4j1 (1.2.17) so it will not impact.*



Please go through the following link for affected apache projects.

https://blogs.apache.org/security/entry/cve-2021-44228

On Thu, Dec 16, 2021 at 4:25 PM Rupert Mazzucco 
wrote:

> The hadoop.apache.org page is curiously silent about this, and there is
> no CVE. Isn't this library used in Hadoop? Pretty sure I saw
> log4j.properties somewhere. Can anybody shed some light on the
> vulnerability of a Hadoop installation? Can it be exploited via RPC? The
> HDFS or YARN web interface? The command line?
>
> Thanks
> Rupert
>
>

-- 



--Brahma Reddy Battula

Re: Unsubscribe

[Brahma Reddy Battula]

Please drop mail to “user-unsubscr...@hadoop.apache.org”…



On Mon, 5 Jul 2021 at 10:27 AM, Madhav A  wrote:

> --



--Brahma Reddy Battula

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Wei-Chiu,

Thanks for driving this..


Looks index.html is not updated and Changelog is broken.


On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
> <https://hadoop.apache.org/docs/r3.3.1/index.html> since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>


-- 



--Brahma Reddy Battula

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Wei-Chiu,

Thanks for driving this..


Looks index.html is not updated and Changelog is broken.


On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
> <https://hadoop.apache.org/docs/r3.3.1/index.html> since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>


-- 



--Brahma Reddy Battula

Re: After deleting data of Hbase table hdfs size is not decreasing HDFS-15812

[Brahma Reddy Battula]

Have a look at *namenode audit logs* after you delete the table which can
tell whether requests reached HDFS or not.


Looks like you are using *"hdp 3.1.4.0-315"* which might not completely *Apache
Hadoop*..SO, as it's vendor specific  , you can ask vendor forum also..



On Thu, Feb 11, 2021 at 4:45 PM Venkatesulu Guntakindapalli
 wrote:

> Hi,
>
> can you check whether snapshots had taken on hbase table in the past,
>
> On Thu, Feb 11, 2021 at 4:14 PM Ayush Saxena  wrote:
>
>> Not sure how hbase or pheonix handle stuff, but do you see the
>> directory/file deleted in HDFS, can check if the file you are deleting is
>> getting deleted, i.e It exists before and once you execute your stuff it
>> isn’t there,
>>
>> Are HDFS Snapshots enabled? Not on the directory but any of its parent
>> also? You can do lsSnapshottableDir, from a super user and check, or some
>> better way, may be Namenode UI used to show
>>
>> Is trash enabled?
>>
>> Can you get the Audit log entry for the file you deleted.
>>
>> If still things are perfect can you get the blocks and DN locations of
>> the file before deleting, FSCK should help here, and check using the block
>> ids using FSCK. If the blocks persist.
>>
>>
>> -Ayush
>>
>> On 11-Feb-2021, at 12:14 PM, satya prakash gaurav 
>> wrote:
>>
>> 
>> Hi Team,
>> Can anyone please help on this issue?
>>
>> Regards,
>> Satya
>>
>> On Wed, Feb 3, 2021 at 7:27 AM satya prakash gaurav 
>> wrote:
>>
>>> Hi Team,
>>>
>>> I have raised a jira HDFS-15812
>>> We are using the hdp 3.1.4.0-315 and hbase 2.0.2.3.1.4.0-315.
>>>
>>> We are deleting the data with normal hbase delete command and even with
>>> api using phoenix. The count is reducing on phoenix and hbase but the
>>> Hdfs size of the hbase directory is not reducing even I ran the major
>>> compaction.
>>>
>>> Regards,
>>> Satya
>>>
>>>
>>>
>>
>> --
>> --
>> Regards,
>> S.P.Gaurav
>>
>>
>
> --
> Thanks & Regards,
>
> Venkatesh
> SRE, Media.net (Data Platform)
> Flock Id- venkatesul...@media.net
> Contact - 9949522101
>
>

-- 



--Brahma Reddy Battula

Re: Mandarin Hadoop online sync this week

[Brahma Reddy Battula]

HI,

what you are planning for this week?

On Tue, Aug 25, 2020 at 10:18 PM Wei-Chiu Chuang  wrote:

> Hello,
>
> There hasn't been a Mandarin online sync for quite some time. I'd like to
> call for one this week:
>
> Date/time:
>
> 8/27 Thursday Beijing Time 1PM
> 8/26 Wednesday US Pacific Time 10PM
>
> Link:
> https://cloudera.zoom.us/j/880548968
>
> Past sync summary:
>
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit
>


-- 



--Brahma Reddy Battula

[ANNOUNCE] Apache Hadoop 3.3.0 release

[Brahma Reddy Battula]

Greetings all,

It gives me great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.3.0.

Apache Hadoop 3.3.0 is the first release of Apache Hadoop 3.3 line for the
year 2020,
which includes 2148 fixes since the previous Hadoop 3.2.0 release.


Of these fixes:
   - 525 in Hadoop Common
   - 804 in HDFS
   - 763 in YARN
   - 56 in MapReduce

Apache Hadoop 3.3.0 contains a number of significant features and
enhancements.A few of them are noted as below.


- *Support ARM:* This is the first release to support ARM architectures

- *Upgrade protobuf from 2.5.0 : *Protobuf upgraded to 3.7.1 as
protobuf-2.5.0 reached EOL.

- *S3A Enhancements : *Lots of enhancements to the S3A code including
Delegation Token support, better handling of 404 caching, S3guard
performance, resilience improvements

- *ABFS Enhancements :* Address issues which surface in the field and tune
things which need tuning, add more tests where appropriate.Improve docs,
especially troubleshooting.

- *Tencent Cloud COS File System Implementation: *Tencent cloud is top
2 cloud vendors in China market and the object store COS is widely
used among China’s cloud users. COSN filesytem to support Tencent
cloud COS natively in Hadoop.

-  *Java11 Runtime Support* : Java 11 runtime support is completed.

- *HDFS RBF stabilizatio**n*: HDFS Router now supports security. Also
contains many bug fixes and improvements.

- *DNS Resolution to support Nameservices to IP Support:*DFS clients can
use a single domain name to discover servers (namenodes/routers/observers)
instead of explicitly listing out all hosts in the config

*- Scheduling of opportunistic containers :  S*cheduling of opportunistic
container through the central RM (YARN-5220), through distributed
scheduling (YARN-2877), as well as the scheduling of containers based on
actual node utilization (YARN-1011) and the container
promotion/demotion (YARN-5085).

*- Application Catalog for YARN applications: **Application catalog
system which provides an editorial and search interface for YARN
applications.*This improves the usability of YARN for managing the
life cycle of applications.

* For major changes included in Hadoop 3.3 line, please refer to Hadoop3.3.0
main page [1].


* For more details about fixes in 3.3.0 release, please read the CHANGELOG
[2] and RELEASENOTES [3].

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [4].

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! This release is a direct result of your great
contributions.

Many thanks to Vinayakumar B who helped in this release process.

[1] https://hadoop.apache.org/docs <https://hadoop.apache.org/docs/r3.2.0/>
[2]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGELOG.3.3.0.html
[3]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html
[4] https://hadoop.apache.org/rele <https://hadoop.apache.org/releases.html>
ases.html





Regards,
Brahma Reddy Battula

Re: Hadoop Jira Bug Workflows/Lifecycle Documentation

[Brahma Reddy Battula]

Hi Ajay,

Good to hear from you, go through the following which help you.

https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

On Mon, May 18, 2020 at 12:48 PM Ajay Bakhshi  wrote:

> Hi Friends,
>
> Does anyone know if there is documentation on How Hadoop Engineering uses
> Jira. Various workflows e.g.
>
> Values of "Status" & "Resolution" field and their meanings.
> How is a bug targeted for multiple releases.
>
> Looking for detailed documentation, if any.
>
> Thanks
>
> -Ajay Bakhshi
>


-- 



--Brahma Reddy Battula

Re: hadoop 3.3.0 release plan

[Brahma Reddy Battula]

Yes, It's supposed to be on March-15th.Mostly this month end, we can start
release voting.

Earlier Plan:

https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0

On Mon, Mar 23, 2020 at 8:49 AM wangyongqiang0...@163.com <
wangyongqiang0...@163.com> wrote:

> is there any release plan for hadoop 3.3.0
>
> thanks
> --
> wangyongqiang0...@163.com
>


-- 



--Brahma Reddy Battula

RE: hadoop-2.8.5 Container launch failed for container_1560691739419_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

[Brahma Reddy Battula]

Looks there is typo in configuration..?  should be 
yarn.nodemanager.aux-services.. Upload yarn-site.xml and 
nodemanager logs


mapreduceyarn.nodemanager.aux-services
mapreduce_shuffle



From: Mike IT Expert [mailto:mikeitexp...@gmail.com]
Sent: Sunday, June 16, 2019 8:30 PM
To: Wanqiang Ji 
Cc: user 
Subject: Re: hadoop-2.8.5 Container launch failed for 
container_1560691739419_0001_01_02 : 
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The 
auxService:mapreduce_shuffle does not exist

I am not sure how to do that so I did buy stop and start scripts as below:

Please let me know how, I am new to this even a link reference wold be very 
helpful.

Thanks

[Selection_054.png]

hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
localhost: nodemanager did not stop gracefully after 5 seconds: killing with 
kill -9
no proxyserver to stop
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ stop-dfs.sh
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-namenode-mike-laptop.out
localhost: starting datanode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-datanode-mike-laptop.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to 
/home/hadoop/hadoop-2.8.5/logs/hadoop-hadoop-secondarynamenode-mike-laptop.out
hadoop@mike-laptop:/tmp/kmeans/kmeans2$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to 
/home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-resourcemanager-mike-laptop.out
localhost: starting nodemanager, logging to 
/home/hadoop/hadoop-2.8.5/logs/yarn-hadoop-nodemanager-mike-laptop.out
hadoop@mike-laptop:/tmp/kmeans/kmeans2$



On Sun, Jun 16, 2019 at 10:42 AM Wanqiang Ji 
mailto:wanqiang...@gmail.com>> wrote:
Do you forgot restart the NodeManager service when updated the yarn-site.xml?

On Sun, Jun 16, 2019 at 5:17 PM Mike IT Expert 
mailto:mikeitexp...@gmail.com>> wrote:
Dear All,

I like to run a python mapreduce sample program. I use the below command and I 
get an exception as provided below:

I googled a few solutions they suggest to update yarn-site.xml as below but I 
still get the same exception message.

Please let me know what you think.

Thank you


hadoop-2.8.5/etc/hadoop/yarn-site.xml

as below:



mapreduceyarn.nodemanager.aux-services
mapreduce_shuffle

  
yarn.nodemanager.aux-services.mapreduce_shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
  

Command and Output

hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh
19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated, please 
use generic option -files instead.
packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py, 
/tmp/kmeans/line_cust/lineCustReduceGroup.py, 
/tmp/hadoop-unjar6079850917434748483/] [] /tmp/streamjob3876527625248696490.jar 
tmpDir=null
19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at 
/0.0.0.0:8032
19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at 
/0.0.0.0:8032
19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to process : 1
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2
19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
job_1560691739419_0001
19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application 
application_1560691739419_0001
19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job: 
http://localhost:8088/proxy/application_1560691739419_0001/
19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001
19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running in 
uber mode : false
19/06/16 14:32:08 INFO mapreduce.Job:  map 0% reduce 0%
19/06/16 14:32:09 INFO mapreduce.Job: Task Id : 
attempt_1560691739419_0001_m_00_0, Status : FAILED
Container launch failed for container_1560691739419_0001_01_02 : 
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The 
auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
at 

Re:

[Brahma Reddy Battula]

By default, all built-in erasure coding policies are disabled, except the
one defined in dfs.namenode.ec.system.default.policy which is enabled by
default(RS-6-3-1024k).  With this configuration, the default EC policy will
be used when no policy name is passed as an argument in the ‘-setPolicy’
command.

You can enable set of policies through hdfs ec [-enablePolicy -policy
] command based on the size of the cluster and the desired
fault-tolerance properties.

For instance, for a cluster with 9 racks, a policy like RS-10-4-1024k will
not preserve rack-level fault-tolerance, and RS-6-3-1024k or RS-3-2-1024kmight
be more appropriate.

Reference:

https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html

On Sun, Feb 17, 2019 at 8:53 PM Shuubham Ojha 
wrote:

> Hello, I am trying to use Hadoop 3.1.1 on my cluster. I wish to experiment
> with the Hitchhiker Code which I believe was introduced in Hadoop 3 itself.
> I don't understand how do I activate the hitchhiker feature for the blocks
> of files I put on the datanode. I also don't know which erasure coding
> policy is being used by default on the uploaded blocks of files when I
> don't do anything. Any help regarding setting the erasure coding policy
> (and hitchhiker feature) would be appreciated.
>
> It's a bit urgent.
>
> Warm regards,
> Shuubham Ojha
>


-- 



--Brahma Reddy Battula

Re: Could not run dfsadmin command with -getStoragePolicy or -setStoragePolicy.

[Brahma Reddy Battula]

Please refer following doc for storage policy commands.now it’s “hdfs
storagepolicies”

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html


Regards
Brahmareddy

On Wed, Sep 26, 2018 at 12:05 PM, Kousuke Miyamoto <
k-miyam...@ze.nes.jp.nec.com> wrote:

> Hi users,
> I use HDFS version 2.7.3 with that command manual but I could not run
> dfsadmin command with -getStoragePolicy or -setStoragePolicy.
>
> $ hdfs dfsadmin -getStoragePolicy /user/guest/
> getStoragePolicy: Unknown command
>
> $ hdfs dfsadmin -setStoragePolicy /user/guest/ COLD
> setStoragePolicy: Unknown command
>
> Do these options exist in dfsadmin command?
> If not, is this a manual bug?
>
> Reference:<
> https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin
> >
>
> Best Regards
>
> Kousuke Miyamoto
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: user-h...@hadoop.apache.org
>
> --



--Brahma Reddy Battula

RE: Security problem extra

[Brahma Reddy Battula]

Unknowingly "dfs.encrypt.data.transfer" configured as true in datanode??
Please cross with check datanode configurations using the following way.
http://:/conf

bq.1 what the client mean, it mean the application running on hdfs, how does it 
have a encryption?
Yes, application (client) which is connected to DN while handshake

From: ZongtianHou [mailto:zongtian...@icloud.com]
Sent: 27 June 2018 14:54
To: user@hadoop.apache.org
Subject: Re: Security problem extra

Does anyone have some clue about it? I have updated the jdk, and still cannot 
solve the problem. Thx advance for any info!!
On 27 Jun 2018, at 12:23 AM, ZongtianHou 
mailto:zongtian...@icloud.com>> wrote:

This is the log info: org.apache.hadoop.hdfs.server.datanode.DataNode: Failed 
to read expected encryption handshake from client at /127.0.0.1:53611. Perhaps 
the client is running an  older version of Hadoop which does not support 
encryption

I have two more questions here.
1 what the client mean, it mean the application running on hdfs, how does it 
have a encryption?
2 I have turn off the encryption about data transfer, rpc protection, http 
protection by setting properties of  hadoop.rpc.protection, 
dfs.encrypt.data.transfer and dfs.http.policy as false, why there is still 
encryption?

Any clue will be appreciated.

RE: No FileSystem for scheme: hdfs when using hadoop-2.8.0 jars

[Brahma Reddy Battula]

Looks jar(hadoop-hdfs-2.8.0.jar) is missing in the classpath.Please check the 
client classpath.

Might be there are no permissions OR missed the this jar while copying..?


Reference:

org.apache.hadoop.fs.FileSystem#getFileSystemClass

if (clazz == null) {
  throw new UnsupportedFileSystemException("No FileSystem for scheme "
  + "\"" + scheme + "\"");
}






--Brahma Reddy Battula

From: omprakash [mailto:ompraka...@cdac.in]
Sent: 31 July 2017 18:17
To: 'user'
Subject: No FileSystem for scheme: hdfs when using hadoop-2.8.0 jars

Hi all,

I have moved my Hadoop-2.7.0  cluster to 2.8.0 version.  I have a client 
application that uses hdfs to get and store file. But after replacing the 2.7.0 
jars with new jars(version 2.8.0) I am facing below exception

Exception in thread "main" java.io.IOException: No FileSystem for scheme: hdfs
  at 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2798)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2809)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
  at 
HadoopTestStubs.HadoopHighAvailabilityTest.main(HadoopHighAvailabilityTest.java:31)

Below is the  code I am trying to execute

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class HadoopHighAvailabilityTest {
public static void main(String[] args) throws IOException {

Configuration conf = new Configuration(false);
conf.set("fs.defaultFS", "hdfs://HdfsCluster");
conf.set("fs.default.name", conf.get("fs.defaultFS"));
conf.set("dfs.nameservices","HdfsCluster");
conf.set("dfs.ha.namenodes.HdfsCluster", "namenode1,namenode2");
conf.set("dfs.namenode.rpc-address.HdfsCluster.namenode1","node1:8020");
conf.set("dfs.namenode.rpc-address.HdfsCluster.namenode2", 
"node2:8020");

conf.set("dfs.client.failover.proxy.provider.HdfsCluster","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

printConfigurations(conf);
FileSystem  fs = FileSystem.get(conf);  // Exception here

}
}


What am I doing wrong here? I have checked the core-site.xml file for release 
2.8.0. I can see there are change in FileSystem implementation but couldn't 
figure out why above code is not working.

Please help.


Regards
Omprakash Paliwal


---
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
---

Re: Hadoop Issues - Can't find on StackOverflow

[Brahma Reddy Battula]

I guess it's in DOS format..can you execute "dos2unix start-yarn.sh" and check 
once..?



From: johnsonngu...@ups.com.INVALID 
Sent: Wednesday, July 19, 2017 3:06 AM
To: user@hadoop.apache.org
Subject: Hadoop Issues - Can't find on StackOverflow


Hi Hadoop People,



 I cannot get my YARN to run for my single node cluster. The error I 
receive when I run start-yarn.sh is:

No Title

starting yarn daemons

mkdir: cannot create directory ‘/opt/hadoop/bin/yarn.cmd’: Not a directory

chown: cannot access ‘/opt/hadoop/bin/yarn.cmd/logs’: Not a directory

starting resourcemanager, logging to 
/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out

/root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 123: cd: /opt/hadoop/bin/yarn.cmd: 
Not a directory

/root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 124: 
/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not a 
directory

head: cannot open 
‘/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out’ for 
reading: Not a directory

/root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 129: 
/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not a 
directory

/root/hadoop-2.7.2/sbin/yarn-daemon.sh: line 130: 
/opt/hadoop/bin/yarn.cmd/logs/yarn-vkq8pyw-resourcemanager-master.out: Not a 
directory





If there is a fix to this, please do tell. Thank you for your help and reading.



Best Regards,

Johnson Nguyen

RE: Namenode not able to come out of SAFEMODE

[Brahma Reddy Battula]

Dn1 and Dn2 is re-started successfully..? can you check the DN logs..?


--Brahma Reddy Battula

From: omprakash [mailto:ompraka...@cdac.in]
Sent: 20 July 2017 12:12
To: user@hadoop.apache.org
Subject: Namenode not able to come out of SAFEMODE

Hi all,

I have a setup of 3 node Hadoop cluster(Hadoop-2.8.0). I have deployed 2 
namenodes that are configured in HA mode using QJM. 2 datanodes are configured 
on the same machine where namenode are installed. 3rd node is used for quorum 
purpose only.

Setup
Node1 -> nn1, dn1, jn1, zkfc1, zkServer1
Node2 -> nn2, dn2, jn2, zkfc2, zkServer2
Node3 -> jn3,  zkServer3

I stopped the cluster for some reason(power recycled the servers)  and since 
them I am not able to start the cluster successfully. After examining the logs 
I found that the namenodes are in safe mode and none of them are able to load 
the block in memory. Below is the status of namenode from namenode UI.


Safe mode is ON. The reported blocks 0 needs additional 6132675 blocks to reach 
the threshold 0.9990 of total blocks 6138814. The number of live datanodes 0 
has reached the minimum number 0. Safe mode will be turned off automatically 
once the thresholds have been reached.

61,56,984 files and directories, 61,38,814 blocks = 1,22,95,798 total 
filesystem object(s).

Heap Memory used 5.6 GB of 7.12 GB Heap Memory. Max Heap Memory is 13.33 GB.

Non Heap Memory used 45.19 MB of 49.75 MB Commited Non Heap Memory. Max Non 
Heap Memory is 130 MB.

I have tried increasing the HADOOP_HEAPSIZE, increasing the heap size in 
HADOOP_NAMENODE_OPTS but no success.
Need help.


Regards
Omprakash Paliwal
HPC-Medical and Bioinformatics Applications Group
Centre for Development of Advanced Computing (C-DAC)
Pune University campus,
PUNE-411007
Maharashtra, India
email:ompraka...@cdac.in
Contact : +91-20-25704231


---
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
---

RE: Unsubscribe

[Brahma Reddy Battula]

It doesn’t like that.. Kindly drop a mail to 
"user-unsubscr...@hadoop.apache.org<mailto:user-unsubscr...@hadoop.apache.org>"



--Brahma Reddy Battula

From: Shawn Du [mailto:shawndow...@gmail.com]
Sent: 17 July 2017 13:30
To: user@hadoop.apache.org
Subject: Unsubscribe

Unsubscribe

Thanks
Shawn

RE: reconfiguring storage

[Brahma Reddy Battula]

Hi Brain Jeltema
1)   Change the data dir configuration
2)   Run dfsadmin -reconfig datanode HOST:PORT start

Reference:
http://hadoop.apache.org/docs/r3.0.0-alpha4/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
  (DataNode Hot Swap Drive)
https://issues.apache.org/jira/browse/HDFS-6727


Note : Decommission might not proper if there are no ecnough nodes in the 
cluster ,it might fail.




--Brahma Reddy Battula

From: Brian Jeltema [mailto:bdjelt...@gmail.com]
Sent: 07 July 2017 22:24
To: user
Subject: Re: reconfiguring storage

I prefer to decommission - reconfigure - recommission.

If hdfs is configured to use volumes at /hdfs-1, /hdfs-2 and /hdfs-3, can I 
just delete the entire
contents of those volumes before recommissioning?

On Jul 6, 2017, at 12:29 PM, daemeon reiydelle 
<daeme...@gmail.com<mailto:daeme...@gmail.com>> wrote:

Another option is to stop the node's relevant Hadoop services (including e.g 
spark, impala, etc. if applicable), move the existing local storage, mount the 
desired file system, and move the data over. Then just restart hadoop. As long 
as this does not take too long, you don't have write consistency that forces 
that shard to be written, etc. you will be fine.



Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Thu, Jul 6, 2017 at 9:17 AM, Brian Jeltema 
<bdjelt...@gmail.com<mailto:bdjelt...@gmail.com>> wrote:
I recently discovered that I made a mistake setting up some cluster nodes and 
didn’t
attach storage to some mount points for HDFS. To fix this, I presume I should 
decommission
the relevant nodes, fix the mounts, then recommission the nodes.

My question is, when the nodes are recommissioned, will the HDFS storage
automatically be reset to ‘empty’, or do I need to perform some sort of explicit
initialization on those volumes before returning the nodes to active status.
-
To unsubscribe, e-mail: 
user-unsubscr...@hadoop.apache.org<mailto:user-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
user-h...@hadoop.apache.org<mailto:user-h...@hadoop.apache.org>

Re: Tentative release date for 2.7.4?

[Brahma Reddy Battula]

AFAIK,Mostly It will be out in couple of weeks.


Please stay tuned following link for more details.


https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4

Hadoop 2.7.4 - Hadoop - Apache Software 
Foundation
cwiki.apache.org
TODO item Status - June 12, 2017 Current Status; All blockers target for 2.7.4 
are landed. (except HADOOP-14356) 1 blockers and 2 critical issues left






From: Swagata Roy 
Sent: Saturday, July 8, 2017 5:06 AM
To: user@hadoop.apache.org
Subject: Tentative release date for 2.7.4?

Hi,

We are looking for a fix for the following issue in our Spring-XD distro -
http://issues.apache.org/jira/browse/HADOOP-13433
[HADOOP-13433] Race in UGI.reloginFromKeytab - ASF 
JIRA
issues.apache.org
It rarely happens, but if it happens, the regionserver will be stuck and can 
never recover. Recently we added a log after a successful re-login which prints 
the ...




Could you please let us know on a tentative timeline for 2.7.4 release?

Thanks,
Swagata

RE: Mismatch in length of source:

[Brahma Reddy Battula]

Distcp can be success with snapshot, but open files length can be zero..? see 
HDFS-11402

AFAIK, if you know the open files you can call recoverlease or wait for 
hardlimit (let Namenode trigger lease recovery). 

i) Get the list of open files

e.g hdfs fsck -openforwrite / -files -blocks -locations | grep -i 
"OPENFORWRITE:"

ii) call recoverylease on each open files

e.g hdfs debug recoverlease

Note: Service like HBase where RS will keep open WAL files, better stop HBase 
service which can automatically close the file.

iii) and then go for distcp



Bytheway,HDFS-10480 gives list of open files.




--Brahma Reddy Battula

-Original Message-
From: Ulul [mailto:had...@ulul.org] 
Sent: 02 January 2017 23:05
To: user@hadoop.apache.org
Subject: Re: Mismatch in length of source:

Hi

I can't remember the exact error message but distcp consistently fails when 
trying to copy open files. Is it your case ?

Workaround it to snapshot prior to copying

Ulul


On 31/12/2016 19:25, Aditya exalter wrote:
> Hi All,
>   A very happy new year to ALL.
>
>   I am facing issue while executing distcp between two different 
> clusters,
>
> Caused by: java.io.IOException: Mismatch in length of 
> source:hdfs://ip1/xx/x and
> target:hdfs://nameservice1/xx/.distcp.tmp.attempt_1483200922993_00
> 56_m_11_2
>
> I tried using -pb and -skipcrccheck
>
>  hadoop distcp -pb -skipcrccheck -update hdfs://ip1/xx/x 
> hdfs:////
>
> hadoop distcp -pb  hdfs://ip1/xx/x hdfs:////
>
> hadoop distcp -skipcrccheck -update
> hdfs://ip1/xx/x hdfs:////
>
>
> but nothing seems to be working .Any solutions please.
>
>
> Regards,
> Aditya.


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

RE: java.io.IOException on Namenode logs

[Brahma Reddy Battula]

Hi Nishant Verma

It will be great, if you mention which version of Hadoop you are using.

Apart from your findings(even I appreciate) and daemeon mentioned, you can 
check following also.


1)  Non-dfs used is more(you can check in namenodeUI/adminreport/jmx)

2)  Scheduled blocks are more(you can check jmx)

If there is any possibility enable the debug logs which can give useful info.


--Brahma Reddy Battula

From: daemeon reiydelle [mailto:daeme...@gmail.com]
Sent: 04 July 2017 01:04
To: Nishant Verma
Cc: user
Subject: Re: java.io.IOException on Namenode logs

A possibility is that the node showing errors was not able to get tcp 
connection, or heavy network conjestion, or (possibly) heavy garbage collection 
tomeouts. Would suspect network
...
There is no sin except stupidity - Oscar Wilde
...
Daemeon (Dæmœn) Reiydelle
USA 1.415.501.0198

On Jul 3, 2017 12:27 AM, "Nishant Verma" 
<nishant.verma0...@gmail.com<mailto:nishant.verma0...@gmail.com>> wrote:
Hello

I am having Kafka Connect writing records on my HDFS nodes. HDFS cluster has 3 
datanodes. Last night I observed data loss in records committed to HDFS. There 
was no issue on Kafka Connect side. However, I can see Namenode showing below 
error logs:

java.io.IOException: File 
/topics/+tmp/testTopic/year=2017/month=07/day=03/hour=03/8237cfb7-2b3d-4d5c-ab04-924c0f647cd6_tmp
 could only be replicated to 0 nodes instead of minReplication (=1).  There are 
3 datanode(s) running and no node(s) are excluded in this operation.
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed 
to place enough replicas, still in need of 3 to reach 3 
(unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, 
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, 
newBlock=true) For more information, please enable DEBUG log level on 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy


Before occurence of every such line, we see below line:
2017-07-02 23:33:43,255 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 
on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 
10.1.2.3:4982<http://10.1.2.3:4982> Call#274492 Retry#0

10.1.2.3 is one of the Kafka Connect nodes.


I checked below things:

- There is no disk issue on datanodes. There is 110 GB space left in each 
datanode.
- In dfsadmin report, there are 3 live datanodes showing.
- dfs.datanode.du.reserved is used as its default value i.e. 0
- dfs.replication is set as 3.
- dfs.datanode.handler.count is used as its default value i.e. 10.
- dfs.datanode.data.dir.perm is used as its default value i.e. 700. But single 
user is used everywhere. So permission issue would not be there. Also, it did 
give accurate result for 22 hours and happened after 22nd hour.
- Could not find any error occurrence for this timestamp in datanode logs.
- The path where dfs.data.dir points has 64% space available on disk.

What could be the cause of this error and how to fix this? Why is it saying the 
file could only be replicated to 0 nodes when it also says there are 3 
datanodes available?

Thanks
Nishant

RE: Unsubscribe

[Brahma Reddy Battula]

It doesn't work like that. Kindly drop a mail to 
"user-unsubscr...@hadoop.apache.org<mailto:user-unsubscr...@hadoop.apache.org>"



--Brahma Reddy Battula

From: Atul Rajan [mailto:atul.raja...@gmail.com]
Sent: 03 July 2017 15:19
To: Donald Nelson
Cc: user@hadoop.apache.org
Subject: Re: Unsubscribe


Unsubscribe

On 3 July 2017 at 12:39, Donald Nelson 
<donald.nel...@uniscon.de<mailto:donald.nel...@uniscon.de>> wrote:

unsubscribe

On 07/03/2017 09:08 AM, nfs_ nfs wrote:

Unsubscribe




--
Best Regards
Atul Rajan

Re: unsubscribe

[Brahma Reddy Battula]

welcome!!


From: Yue Cheng <yue...@gmail.com>
Sent: Sunday, July 2, 2017 3:06 AM
To: Brahma Reddy Battula
Subject: Re: unsubscribe

Thanks Brahma!

Best,
Yue

On Fri, Jun 30, 2017 at 8:20 PM, Brahma Reddy Battula 
<brahmareddy.batt...@hotmail.com<mailto:brahmareddy.batt...@hotmail.com>> wrote:

It doesn't work like that. Kindly drop a mail to 
"user-unsubscr...@hadoop.apache.org<mailto:user-unsubscr...@hadoop.apache.org>"



From: Yue Cheng <yue...@gmail.com<mailto:yue...@gmail.com>>
Sent: Saturday, July 1, 2017 2:47 AM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: unsubscribe

Please sign me off.

Thanks.

Re: unsubscribe

[Brahma Reddy Battula]

It doesn't work like that. Kindly drop a mail to 
"user-unsubscr...@hadoop.apache.org"



From: Yue Cheng 
Sent: Saturday, July 1, 2017 2:47 AM
To: user@hadoop.apache.org
Subject: unsubscribe

Please sign me off.

Thanks.

Re: Ensure High Availability of Datanodes in a HDFS cluster

[Brahma Reddy Battula]

1.Yes, those will ensure that file will be written to available nodes .


2.

BlockManager: defaultReplication = 2

This is the Default block replication which you configured in server 
(Namenode). The actual number of replications can be specified when the file is 
created. The default is used if replication is not specified in create time.



3. "dfs.replication" is client(in your case confluent kafka) side property.May 
be,you can cross check this configuration in kafka.



-Brahma Reddy Battula

From: Nishant Verma <nishant.verma0...@gmail.com>
Sent: Friday, June 30, 2017 7:50 PM
To: common-u...@hadoop.apache.org
Subject: Ensure High Availability of Datanodes in a HDFS cluster


Hi

I have a two master and three datanode HDFS cluster setup. They are AWS EC2 
instances.

I have to test High Availability of Datanodes i.e., if during load run where 
data is written on HDFS, a datanode dies then there is no data loss. The two 
remaning datanodes which are alive should take care of the data writes.

I have set below properties in hdfs-site.xml. dfs.replication = 2 (because if 
any one datanode dies, then there is no issue of not able to meet replication 
factor)

dfs.client.block.write.replace-datanode-on-failure.policy = ALWAYS
dfs.client.block.write.replace-datanode-on-failure.enable = true
dfs.client.block.write.replace-datanode-on-failure.best-effort = true


My questions are:

1 - Does setting up above properties suffice my Datanode High Availability? Or 
something else is needed? 2 - On dfs service startup, I do see below INFO on 
namenode logs:

2017-06-27 10:51:52,546 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication  
   = 2
2017-06-27 10:51:52,546 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication  
   = 512
2017-06-27 10:51:52,546 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication  
   = 1
2017-06-27 10:51:52,546 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: 
maxReplicationStreams  = 2


But I still see that the files being created on HDFS are with replication 
factor 3. Why is that so? This would hurt my High Availability of Datanodes.

-rw-r--r--   3 hadoopuser supergroup 247373 2017-06-29 09:36 
/topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+210+0001557358+0001557452
-rw-r--r--   3 hadoopuser supergroup   1344 2017-06-29 08:33 
/topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432839+0001432850
-rw-r--r--   3 hadoopuser supergroup   3472 2017-06-29 09:03 
/topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+228+0001432851+0001432881
-rw-r--r--   3 hadoopuser supergroup   2576 2017-06-29 08:33 
/topics/testTopic/year=2017/month=06/day=29/hour=14/testTopic+23+0001236477+0001236499


P.S. - My records are written on HDFS by Confluent Kafka Connect HDFS Sink 
Connector.


Thanks

Nishant

Re: Any ideas when Hadoop 3 will become final?

[Brahma Reddy Battula]

Tentative dates are mentioned in following link, it might be available mid of 
August-2017.



https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release

Hadoop 3.0.0 release - Hadoop - Apache Software 
Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release>
cwiki.apache.org
Release schedule. For Hadoop 3, we are planning to "release early, release 
often" to quickly iterate on feedback collected from downstream projects.





-Brahma Reddy Battula


From: Jasson Chenwei <ynjassionc...@gmail.com>
Sent: Tuesday, June 13, 2017 6:11 AM
To: Karthik Shyamsunder
Cc: user
Subject: Re: Any ideas when Hadoop 3 will become final?

Good one. I also want to know!! Also, are there any efforts to make projects 
(e.g., Hive and Spark) in Hadoop-eco to be compatible with Hadoop-3.0 ?


Thanks,

Wei

On Sat, Jun 10, 2017 at 5:58 AM, Karthik Shyamsunder 
<karthik.shyamsun...@gmail.com<mailto:karthik.shyamsun...@gmail.com>> wrote:
Folks,

Any ideas when Hadoop 3 will become final? Its currently Hadoop 3.0.0 alpah3.  
Are we looking at 2017 or 2018 for GA?  Please advise.

Sincerely,

Karthik

RE: HDFS file replication to slave nodes not working

[Brahma Reddy Battula]

Yes, you can configure “dfs.datanode.data.dir”.

Reference for default configurations:
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml


Regards
Brahma Reddy Battula

From: Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
Sent: 14 June 2017 17:47
To: Brahma Reddy Battula
Cc: common-u...@hadoop.apache.org
Subject: Re: HDFS file replication to slave nodes not working

Any way I can tell hadoop to use the /mnt dir instead of 
/tmp/hadoop-{user-name} directory to store the files?

Thanks
Bhushan Pathak

Thanks
Bhushan Pathak

On Wed, Jun 14, 2017 at 3:06 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:


Please see my comments inline.



Regards
Brahma Reddy Battula

From: Bhushan Pathak 
[mailto:bhushan.patha...@gmail.com<mailto:bhushan.patha...@gmail.com>]
Sent: 14 June 2017 17:14
To: common-u...@hadoop.apache.org<mailto:common-u...@hadoop.apache.org>
Subject: HDFS file replication to slave nodes not working

Hello,

I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The 
hdfs-site.xml file has the following config -

dfs.namenode.name.dir
file:/mnt/hadoop_store/datanode


dfs.datanode.name.dir
file:/mnt/hadoop_store/namenode


===> property should be “dfs.datanode.data.dir”. Please have a look following 
for all default configurations.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was 
successful.

My assumption is that the 3 csv files should be present on all 3 nodes, either 
under the datanode or the namenode directory. On the master, I can see the 
following files -

[hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
Found 3 items
-rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34 
/usr/hadoop/Final_Album_file.csv
-rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34 
/usr/hadoop/Final_Artist_file.csv
-rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34 
/usr/hadoop/Final_Tracks_file.csv
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
current  in_use.lock
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
edits_001-002  
edits_027-028  
edits_055-056
edits_003-004  
edits_029-030  
edits_057-058
edits_005-006  
edits_031-032  
edits_059-060
edits_007-008  
edits_033-034  
edits_061-064
edits_009-010  
edits_035-036  
edits_065-096
edits_011-012  
edits_037-038  
edits_inprogress_097
edits_013-014  
edits_039-040  fsimage_064
edits_015-016  
edits_041-042  fsimage_064.md5
edits_017-017  
edits_043-044  fsimage_096
edits_018-019  
edits_045-046  fsimage_096.md5
edits_020-020  
edits_047-048  seen_txid
edits_021-022  
edits_049-050  VERSION
edits_023-024  
edits_051-052
edits_025-026  
edits_053-054
[hadoop@master hadoop-2.7.3]$

While on the 2 slave nodes, there are only empty directories. Is my assumption 
that the 3 csv files should be replicated to slave nodes as well correct? If 
yes, why are  they missing from the slave nodes? Additionally, are the files 
that I see in datanode/current directory of master the actual csv files that I 
have uploaded?


Yes, it will replicate to 3 nodes.(it’s based on “dfs.replication” which is “3” 
by default)
The location which you are checking is wrong, since property is wrong..and by 
default it will stored under “/tmp/Hadoop-${user-name}}.
Data under “datanode/current directory” is meta data for all operations.

Please go through the following design to know more about HDFS.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html


Thanks
Bhushan Pathak

RE: HDFS file replication to slave nodes not working

[Brahma Reddy Battula]

Please see my comments inline.



Regards
Brahma Reddy Battula

From: Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
Sent: 14 June 2017 17:14
To: common-u...@hadoop.apache.org
Subject: HDFS file replication to slave nodes not working

Hello,

I have hadoop 2.7.3 running on a 3-node cluster [1 master, 2 slaves]. The 
hdfs-site.xml file has the following config -

dfs.namenode.name.dir
file:/mnt/hadoop_store/datanode


dfs.datanode.name.dir
file:/mnt/hadoop_store/namenode


===> property should be “dfs.datanode.data.dir”. Please have a look following 
for all default configurations.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

I used the 'hdfs -put' command to upload 3 csv files to HDFS, which was 
successful.

My assumption is that the 3 csv files should be present on all 3 nodes, either 
under the datanode or the namenode directory. On the master, I can see the 
following files -

[hadoop@master hadoop-2.7.3]$ bin/hdfs dfs -ls /usr/hadoop
Found 3 items
-rw-r--r--   3 hadoop supergroup 124619 2017-06-14 14:34 
/usr/hadoop/Final_Album_file.csv
-rw-r--r--   3 hadoop supergroup  68742 2017-06-14 14:34 
/usr/hadoop/Final_Artist_file.csv
-rw-r--r--   3 hadoop supergroup2766110 2017-06-14 14:34 
/usr/hadoop/Final_Tracks_file.csv
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/namenode/
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/
current  in_use.lock
[hadoop@master hadoop-2.7.3]$ ls /mnt/hadoop_store/datanode/current/
edits_001-002  
edits_027-028  
edits_055-056
edits_003-004  
edits_029-030  
edits_057-058
edits_005-006  
edits_031-032  
edits_059-060
edits_007-008  
edits_033-034  
edits_061-064
edits_009-010  
edits_035-036  
edits_065-096
edits_011-012  
edits_037-038  
edits_inprogress_097
edits_013-014  
edits_039-040  fsimage_064
edits_015-016  
edits_041-042  fsimage_064.md5
edits_017-017  
edits_043-044  fsimage_096
edits_018-019  
edits_045-046  fsimage_096.md5
edits_020-020  
edits_047-048  seen_txid
edits_021-022  
edits_049-050  VERSION
edits_023-024  
edits_051-052
edits_025-026  
edits_053-054
[hadoop@master hadoop-2.7.3]$

While on the 2 slave nodes, there are only empty directories. Is my assumption 
that the 3 csv files should be replicated to slave nodes as well correct? If 
yes, why are  they missing from the slave nodes? Additionally, are the files 
that I see in datanode/current directory of master the actual csv files that I 
have uploaded?


Yes, it will replicate to 3 nodes.(it’s based on “dfs.replication” which is “3” 
by default)
The location which you are checking is wrong, since property is wrong..and by 
default it will stored under “/tmp/Hadoop-${user-name}}.
Data under “datanode/current directory” is meta data for all operations.

Please go through the following design to know more about HDFS.
http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html


Thanks
Bhushan Pathak

RE: Yarn not picking up the Resource manager configurations

[Brahma Reddy Battula]

Looks configurations(mainly yarn-site.xml here which you pointed) are not 
loaded in classpath. Can you check the output of “hadoop classpath”..?



Regards
Brahma Reddy Battula

From: Dhanushka Parakrama [mailto:parakrama1...@gmail.com]
Sent: 19 May 2017 16:40
To: user@hadoop.apache.org
Subject: Yarn not picking up the Resource manager configurations

Hi All
I have setup the multi node hadoop cluster . but when is run the simple map 
reduce job it gives me the following error the in  the data nodes container log 
.

Im running the hadoop 2.7.3 version

In namenode which contain resource manager as well . i ran below job

hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /words  /output

Error


tryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:14,340 INFO [main] org.apache.hadoop.ipc.Client: Retrying 
connect to server: 0.0.0.0/0.0.0.0:8030<http://0.0.0.0/0.0.0.0:8030>. Already 
tried 8 time(s); retry policy is 
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-05-19 07:57:15,341 INFO [main] org.apache.hadoop.ipc.Client: Retrying 
connect to server: 0.0.0.0/0.0.0.0:8030<http://0.0.0.0/0.0.0.0:8030>. Already 
tried 9 time(s); retry policy is 
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Configurations
===
vi .barshrc file

export JAVA_HOME=/home/ubuntu/jdk1.8.0_45
export HADOOP_HOME=/usr/local/hadoop/current
#export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper/current

export 
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

vi .yarn-site.xml file







   yarn.nodemanager.aux-services
   mapreduce_shuffle
 


   yarn.nodemanager.aux-services.mapreduce.shuffle.class
   org.apache.hadoop.mapred.ShuffleHandler



yarn.resourcemanager.hostname
nn1.cluster.com<http://nn1.cluster.com>


 
  yarn.resourcemanager.scheduler.address
  nn1.cluster.com:8030<http://nn1.cluster.com:8030>
 





Can any expert help me with this issue , im stuck at this point
Thank You

RE: unsubscribe

[Brahma Reddy Battula]

Kindly send an email to 
user-unsubscr...@hadoop.apache.org


-Brahma

From: shanker valipireddy [mailto:shanker.valipire...@gmail.com]
Sent: 28 April 2017 03:40
To: user-subscr...@hadoop.apache.org; gene...@hadoop.apache.org; user
Subject: unsubscribe



--
Thanks & Regards,
Shanker

RE: Hadoop 2.7.3 cluster namenode not starting

[Brahma Reddy Battula]

Are you sure that you are starting in same machine (master)..?

Please share “/etc/hosts” and configuration files..


Regards
Brahma Reddy Battula

From: Bhushan Pathak [mailto:bhushan.patha...@gmail.com]
Sent: 27 April 2017 17:18
To: user@hadoop.apache.org
Subject: Fwd: Hadoop 2.7.3 cluster namenode not starting

Hello

I have a 3-node cluster where I have installed hadoop 2.7.3. I have updated 
core-site.xml, mapred-site.xml, slaves, hdfs-site.xml, yarn-site.xml, 
hadoop-env.sh files with basic settings on all 3 nodes.

When I execute start-dfs.sh on the master node, the namenode does not start. 
The logs contain the following error -
2017-04-27 14:17:57,166 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: 
Failed to start namenode.
java.net.BindException: Problem binding to [master:51150] 
java.net.BindException: Cannot assign requested address; For more details see:  
http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.net<http://org.apache.hadoop.net>.NetUtils.wrapWithMessage(NetUtils.java:792)
at 
org.apache.hadoop.net<http://org.apache.hadoop.net>.NetUtils.wrapException(NetUtils.java:721)
at org.apache.hadoop.ipc.Server.bind(Server.java:425)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:574)
at org.apache.hadoop.ipc.Server.(Server.java:2215)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:951)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:796)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:345)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:674)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:647)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch<http://sun.nio.ch>.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at 
sun.nio.ch<http://sun.nio.ch>.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:408)
... 13 more
2017-04-27 14:17:57,171 INFO org.apache.hadoop.util.ExitUtil: Exiting with 
status 1
2017-04-27 14:17:57,176 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at master/1.1.1.1<http://1.1.1.1>
/



I have changed the port number multiple times, every time I get the same error. 
How do I get past this?



Thanks
Bhushan Pathak

RE: Request for Hadoop mailing list subscription and 3.0.0 issues

[Brahma Reddy Battula]

Hello Sidharth Kumar

Glad to hear your interest


You can find all the mailing list in following link

https://hadoop.apache.org/mailing_lists.html

Please refer to following  wiki page for more details about how the 
contribution process works.

https://wiki.apache.org/hadoop/HowToContribute



Regards
Brahma Reddy Battula

From: Sidharth Kumar [mailto:sidharthkumar2...@gmail.com]
Sent: 28 March 2017 14:46
To: user@hadoop.apache.org
Subject: Request for Hadoop mailing list subscription and 3.0.0 issues

Hi Folks,

I am working as full time Hadoop Administrator since 2 years and want to be a 
part of apache hadoop foundation so that i can contribute best of my knowledge 
and learn from more about it from experts.So, can you please help me to 
subscribe to other mailing lists.

I also want to know about the issue i was facing while setting apache hadoop 
3.0.0-alpha 2 cluster. I installed and configured the cluster and HDFS was 
working all great but mapreduce was failing. I made multiple attempts by adding 
few more additional configuration but it was looking that job was not able to 
pickup the configurations.While the same set of configuration worked fine for 
hadoop2.7.2 and other stable versions.


Thanks for your help in advance

--
Regards
Sidharth Kumar | Mob: +91 8197 555 599 | 
LinkedIn<https://www.linkedin.com/in/sidharthkumar2792/>

Re: about dfs.datanode.du.reserved

[Brahma Reddy Battula]

You can write a script to update this config.(Even you can manually add
this configuration in hdfs-site.xml of all the datanodes and you can
restart)



--Brahma

On Sun, Feb 12, 2017 at 12:15 PM, Alexis <alz...@gmail.com> wrote:

> Yes you did. Thanks in advance. Is there any way to push this config to
> all the nodes from master or should I make to script to do this?
>
> Regards
>
> Enviado desde mi iPhone
>
> El 12 feb. 2017, a las 02:30, Brahma Reddy Battula <bra...@apache.org>
> escribió:
>
> Hi Alexis Fidalgo
>
> 1) I did not seen this query recently
>
> 2) you need to configure this property in slaves ( DataNode).
>
> *dfs.datanode.du.**reserved : *The number of bytes will be left free on
> the volumes used by the DataNodes. By Default,it's zero.
>
>
> For example if the disk capacity is 1TB and *dfs.datanode.du.**reserved *c
> onfigured with* 100GB.*So DataNode will not use 100GB for block
> allocation,so this data can be used by nodemanager intermittent files,
> log files .
>
> May be you can plan your MR jobs accordingly this. Hope I cleared your
> doubts.
>
> On Sat, Feb 11, 2017 at 7:26 PM, Alexis Fidalgo <alz...@gmail.com> wrote:
>
>> Hello, i’ve tried to search archives (and google) regarding this issue
>> but had no luck. After some changes in our mapreduce code, it takes all the
>> available disk space on datanodes, before this change we hade no problem at
>> all, but since then, every few days, disks on datanodes (we have 4, all
>> with same configuration regarding disk, memory, processor, OS) becomes full
>> and we have no more mapreduce jobs completed. so i need to wipe datanodes
>> and format namenode and start all over again.
>>
>> Reading documentation i found this configuration for hdfs-site.xml
>>
>> 
>> dfs.datanode.du.reserved
>> 32212254720
>> 
>> 
>>
>> Questions regarding this
>>
>> 1. is there any thread already on this issue to read and not to ask again
>> about it?
>> 2. if not 1, do i need to set up this property only on master or every
>> slave too?
>> 3. will this fix the problem or just avoid the disk become full but the
>> MR jobs will fail the same (no more space to work so we need to review our
>> code)
>>
>>
>> thanks in advance, sorry if im asking about an already discussed issue, i
>> just suscribed to the list.
>>
>> regards
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: user-h...@hadoop.apache.org
>>
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>
>


-- 



--Brahma Reddy Battula

Re: about dfs.datanode.du.reserved

[Brahma Reddy Battula]

Hi Alexis Fidalgo

1) I did not seen this query recently

2) you need to configure this property in slaves ( DataNode).

*dfs.datanode.du.**reserved : *The number of bytes will be left free on the
volumes used by the DataNodes. By Default,it's zero.


For example if the disk capacity is 1TB and *dfs.datanode.du.**reserved *
configured with* 100GB.*So DataNode will not use 100GB for block
allocation,so this data can be used by nodemanager intermittent files, log
files .

May be you can plan your MR jobs accordingly this. Hope I cleared your
doubts.

On Sat, Feb 11, 2017 at 7:26 PM, Alexis Fidalgo <alz...@gmail.com> wrote:

> Hello, i’ve tried to search archives (and google) regarding this issue but
> had no luck. After some changes in our mapreduce code, it takes all the
> available disk space on datanodes, before this change we hade no problem at
> all, but since then, every few days, disks on datanodes (we have 4, all
> with same configuration regarding disk, memory, processor, OS) becomes full
> and we have no more mapreduce jobs completed. so i need to wipe datanodes
> and format namenode and start all over again.
>
> Reading documentation i found this configuration for hdfs-site.xml
>
> 
> dfs.datanode.du.reserved
> 32212254720
> 
> 
>
> Questions regarding this
>
> 1. is there any thread already on this issue to read and not to ask again
> about it?
> 2. if not 1, do i need to set up this property only on master or every
> slave too?
> 3. will this fix the problem or just avoid the disk become full but the MR
> jobs will fail the same (no more space to work so we need to review our
> code)
>
>
> thanks in advance, sorry if im asking about an already discussed issue, i
> just suscribed to the list.
>
> regards
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: user-h...@hadoop.apache.org
>
>


-- 



--Brahma Reddy Battula

RE: moving hadoop and hive tables with its table structure and schema to new hadoop cluster

[Brahma Reddy Battula]

Hi

Distcp can be used for same…DistCp is a tool used for large inter/intra-cluster 
copying.



References:

http://hadoop.apache.org/docs/stable/hadoop-distcp/DistCp.html
https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_admin_distcp_data_cluster_migrate.html
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_administration/content/using_distcp.html


Regards
Brahma Reddy Battula

From: Sirisha Cheruvu [mailto:sirisha5...@gmail.com]
Sent: 10 January 2017 19:15
To: user@hadoop.apache.org
Subject: moving hadoop and hive tables with its table structure and schema to 
new hadoop cluster

Hi,

How to move all hadoop data  including hive tables and its metastore tables 
from one cluster into new cluster.. I have to migrate all data from staging 
cluster to production cluster..


Regards,
Sirisha Ch,
Phone:+91-9611482943

RE: Hadoop source code modification.

[Brahma Reddy Battula]

Following links might be useful fro you.


https://wiki.apache.org/hadoop/EclipseEnvironment
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
https://www.quora.com/What-are-the-best-ways-to-learn-about-Hadoop-source


Regards
Brahma Reddy Battula

From: Madhvaraj Shetty [mailto:madresearch2...@gmail.com]
Sent: 15 November 2016 20:27
To: Brahma Reddy Battula
Subject: Re: Hadoop source code modification.

Thank you so much sir
Any links for understanding hadoop code and classes?
Any quick way to debug OR any setup like eclipse.. to do these work?



On Tue, Nov 15, 2016 at 5:49 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:
(Keeping user-mailing list in loop.)

You can compile corresponding module which you modified.

Please refer  "Where to run Maven from?"  from the following.

https://github.com/apache/hadoop/blob/trunk/BUILDING.txt



Regards
Brahma Reddy Battula

-Original Message-
From: Madhvaraj Shetty 
[mailto:madresearch2...@gmail.com<mailto:madresearch2...@gmail.com>]
Sent: 15 November 2016 19:36
To: gene...@hadoop.apache.org<mailto:gene...@hadoop.apache.org>
Subject: Hadoop source code modification.

Hello,

I am working in the area of hadoop, especially i want to improve hadoop 
security.

So i need procedure  for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part of 
the code.

Is it compulsory to compile the whole hadoop, can i compile/debug a part of it. 
Any steps for that.


Thank you.

RE: Hadoop source code modification.

[Brahma Reddy Battula]

(Keeping user-mailing list in loop.)

You can compile corresponding module which you modified.

Please refer  "Where to run Maven from?"  from the following.

https://github.com/apache/hadoop/blob/trunk/BUILDING.txt



Regards
Brahma Reddy Battula

-Original Message-
From: Madhvaraj Shetty [mailto:madresearch2...@gmail.com] 
Sent: 15 November 2016 19:36
To: gene...@hadoop.apache.org
Subject: Hadoop source code modification.

Hello,

I am working in the area of hadoop, especially i want to improve hadoop 
security.

So i need procedure  for modifying hadoop source code.
I have visited this <https://wiki.apache.org/hadoop/HowToContribute>and
able to compile hadoop from the source code, but i need to modify the part of 
the code.

Is it compulsory to compile the whole hadoop, can i compile/debug a part of it. 
Any steps for that.


Thank you.

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

RE: HDFS - Corrupt replicas preventing decommissioning?

[Brahma Reddy Battula]

Please check my inline comments to your queries. Hope I have answered all your 
questions…


Regards
Brahma Reddy Battula

From: Hariharan [mailto:hariharan...@gmail.com]
Sent: 15 November 2016 18:55
To: user@hadoop.apache.org
Subject: HDFS - Corrupt replicas preventing decommissioning?

Hello folks,
I'm running Apache Hadoop 2.6.0 and I'm seeing a weird problem where I keep 
seeing corrupt replicas. Example:
2016-11-15 06:42:38,104 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block: 
blk_1073747320_231160{blockUCState=COMMITTED, primaryNodeIndex=0, 
replicas=[ReplicaUnderConstruction[[DISK]DS-11d5d492-a608-4bc0-9a04-048b8127bb32:NORMAL:10.0.8.185:50010|RBW]]},
 Expected Replicas: 2, live replicas: 0, corrupt replicas: 2, decommissioned 
replicas: 1, excess replicas: 0, Is Open File: true, Datanodes having this 
block: 10.0.8.185:50010<http://10.0.8.185:50010> 
10.0.8.148:50010<http://10.0.8.148:50010> 
10.0.8.149:50010<http://10.0.8.149:50010> , Current Datanode: 
10.0.8.185:50010<http://10.0.8.185:50010>, Is current datanode decommissioning: 
true
But I can't figure out which file this block belongs to - hadoop fsck / -files 
-blocks -locations | grep blk_1073747320_231160 returns nothing.
>> Looks files are open state, you can check fsck with –openforwrite option 
>> which will list all the open files also.
So I'm unable to delete the file and my concern is that this seems to be 
blocking decommissioning of my datanode (going on for ~18 hours now) since, 
looking at the code in BlockManager.java, we would not mark the DN as 
decommissioned if there are blocks with no live replicas on it.
My questions are:
1. What causes corrupt replicas and how to avoid them? I seem to be seeing 
these frequently:
(examples from prior runs)
>>As files are open state, there are chances blocks can be corrupt state since 
>>might not send block received command to Namenode.
So before going for decommission ensure that files are closed and check the 
under-replicated block count.

hadoop-hdfs-namenode-ip-10-0-8-199.log.9:2016-11-13 23:54:57,513 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block: 
blk_1074063633_2846521{blockUCState=COMMITTED, primaryNodeIndex=0, 
replicas=[ReplicaUnderConstruction[[DISK]DS-7b8e7b76-6066-43fb-8340-d93f7ab9c6ea:NORMAL:10.0.8.75:50010|RBW]]},
 Expected Replicas: 2, live replicas: 0, corrupt replicas: 4, decommissioned 
replicas: 1, excess replicas: 0, Is Open File: true, Datanodes having this 
block: 10.0.8.75:50010<http://10.0.8.75:50010> 
10.0.8.156:50010<http://10.0.8.156:50010> 
10.0.8.188:50010<http://10.0.8.188:50010> 
10.0.8.34:50010<http://10.0.8.34:50010> 10.0.8.74:50010<http://10.0.8.74:50010> 
, Current Datanode: 10.0.8.75:50010<http://10.0.8.75:50010>, Is current 
datanode decommissioning: true
hadoop-hdfs-namenode-ip-10-0-8-199.log.9:2016-11-13 23:54:57,513 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block: 
blk_1073975974_2185091{blockUCState=COMMITTED, primaryNodeIndex=0, 
replicas=[ReplicaUnderConstruction[[DISK]DS-b9b8b191-f8c8-49b0-b4c1-b2a9ce6b9ee8:NORMAL:10.0.8.153:50010|RBW]]},
 Expected Replicas: 2, live replicas: 0, corrupt replicas: 3, decommissioned 
replicas: 1, excess replicas: 0, Is Open File: true, Datanodes having this 
block: 10.0.8.153:50010<http://10.0.8.153:50010> 
10.0.8.74:50010<http://10.0.8.74:50010> 10.0.8.7:50010<http://10.0.8.7:50010> 
10.0.8.198:50010<http://10.0.8.198:50010> , Current Datanode: 
10.0.8.153:50010<http://10.0.8.153:50010>, Is current datanode decommissioning: 
true
hadoop-hdfs-namenode-ip-10-0-8-199.log.9:2016-11-13 23:54:57,513 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Block: 
blk_1073975974_2185091{blockUCState=COMMITTED, primaryNodeIndex=0, 
replicas=[ReplicaUnderConstruction[[DISK]DS-b9b8b191-f8c8-49b0-b4c1-b2a9ce6b9ee8:NORMAL:10.0.8.153:50010|RBW]]},
 Expected Replicas: 2, live replicas: 0, corrupt replicas: 3, decommissioned 
replicas: 1, excess replicas: 0, Is Open File: true, Datanodes having this 
block: 10.0.8.153:50010<http://10.0.8.153:50010> 
10.0.8.74:50010<http://10.0.8.74:50010> 10.0.8.7:50010<http://10.0.8.7:50010> 
10.0.8.198:50010<http://10.0.8.198:50010> , Current Datanode: 
10.0.8.7:50010<http://10.0.8.7:50010>, Is current datanode decommissioning: true
2. Is this possibly a JIRA that's fixed in recent versions (I realize I'm 
running a very old version)?
>> Based on the exact root cause for corrupt, we can able to tell jira 
>> Id’s..Need to check all of your logs.
3. Anything I can do to "force" decommissioning of such nodes (apart from 
forcefully terminating them)?
>> As of now no “forceful” decommission. But you can delete the corrupt blocks 
>> using  “hdfs fsck delete ”
Thanks,
Hari

RE: unsubscribe

[Brahma Reddy Battula]

Drop a mail to user-unsubscr...@hadoop.apache.org





--Brahma Reddy Battula

From: Chen Qiming [mailto:qimin...@usc.edu]
Sent: 24 October 2016 14:53
To: user@hadoop.apache.org
Subject: unsubscribe

unsubscribe

RE: LeaseExpiredException: No lease on /user/biadmin/analytic‐root/SX5XPWPPDPQH/.

[Brahma Reddy Battula]

Can you trace namenode logs, whether this is file deleted/renamed(might be 
parent folder) before this reducer run..?




--Brahma Reddy Battula

From: Zhang Jianfeng [mailto:jzhang...@gmail.com]
Sent: 18 October 2016 18:55
To: Gaurav Kumar
Cc: user.hadoop; Rakesh Radhakrishnan
Subject: Re: LeaseExpiredException: No lease on 
/user/biadmin/analytic‐root/SX5XPWPPDPQH/.

Thanks Gaurav. For my case, I called the HDFS API to write the reducer result 
into HDFS directly, not using Spark.

2016-10-17 23:24 GMT+08:00 Gaurav Kumar 
<gauravkuma...@gmail.com<mailto:gauravkuma...@gmail.com>>:

Hi,

Please also check for coalesced RDD. I encountered the same error while writing 
a coalesced rdd/df to HDFS. If this is the case, please use repartition instead.

Sent from OnePlus 3

Thanks & Regards,
Gaurav Kumar

On Oct 17, 2016 11:22 AM, "Zhang Jianfeng" 
<jzhang...@gmail.com<mailto:jzhang...@gmail.com>> wrote:
Thanks Rakesh for your kind help. Actually during the job only one reducer 
result file (for example part-r-2) had this error, other reducers worked well.

Best Regards,
Jian Feng

2016-10-17 11:49 GMT+08:00 Rakesh Radhakrishnan 
<rake...@apache.org<mailto:rake...@apache.org>>:
Hi Jian Feng,

Could you please check your code and see any possibilities of simultaneous 
access to the same file. Mostly this situation happens when multiple clients 
tries to access the same file.

Code Reference:- 
https://github.com/apache/hadoop/blob/branch-2.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L2737

Best Regards,
Rakesh
Intel

On Mon, Oct 17, 2016 at 7:16 AM, Zhang Jianfeng 
<jzhang...@gmail.com<mailto:jzhang...@gmail.com>> wrote:
Hi ,

I hit an wired error. On our hadoop cluster (2.2.0), occasionally a 
LeaseExpiredException is thrown.

The stacktrace is as below:


org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 No lease on /user/biadmin/analytic‐root/SX5XPWPPDPQH/.executions/.at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2737)

at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2801)

at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2783)

at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.com<http://namenode.NameNodeRpcServer.com>plete(NameNodeRpcServer.java:611)

at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:428)

at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59586)

at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

at java.security.AccessController.doPrivileged(AccessController.java:310)

at 
javax.security.auth.Subject.do<http://javax.security.auth.Subject.do>As(Subject.java:573)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)

at org.apache.hadoop.ipc.Client.call(Client.java:1300)

at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

at $Proxy7.complete(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

at java.lang.reflect.Method.invoke(Method.java:611)

at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)

at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at $Proxy7.complete(Unknown Source)

at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371)

at 
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:1894)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1881)

at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)

at java.io.FilterOutputStream.close(FilterOutputStream.java:154)
Any help will be appreciated!

--
Best Regards,
Jian Feng




--
Best Regards,
Jian Feng




--
Best Regards,
Jian Feng

RE: hdfs2.7.3 kerberos can not startup

[Brahma Reddy Battula]

Seems to be property problem.. it should be principal ( “l” is missed).


  dfs.secondary.namenode.kerberos.principa
  hadoop/_h...@example.com<mailto:h...@example.com>



For namenode httpserver start fail, please check rakesh comments..

This is probably due to some missing configuration.
Could you please re-check the ssl-server.xml, keystore and truststore 
properties:

ssl.server.keystore.location
ssl.server.keystore.keypassword
ssl.client.truststore.location
ssl.client.truststore.password


--Brahma Reddy Battula

From: kevin [mailto:kiss.kevin...@gmail.com]
Sent: 20 September 2016 16:53
To: Rakesh Radhakrishnan
Cc: user.hadoop
Subject: Re: hdfs2.7.3 kerberos can not startup

thanks, but my issue is name node could  Login successful,but second namenode 
couldn't. and name node got a HttpServer.start() threw a non Bind IOException:

hdfs-site.xml:


dfs.webhdfs.enabled
true



  dfs.block.access.token.enable
  true




  dfs.namenode.kerberos.principal
  hadoop/_h...@example.com<mailto:h...@example.com>


  dfs.namenode.keytab.file
  /etc/hadoop/conf/hdfs.keytab


  dfs.https.port
  50470


  dfs.namenode.https-address
  dmp1.example.com:50470<http://dmp1.example.com:50470>


  dfs.namenode.kerberos.internal.spnego.principa
  HTTP/_h...@example.com<mailto:h...@example.com>


  dfs.web.authentication.kerberos.keytab
  /etc/hadoop/conf/hdfs.keytab


  dfs.http.policy
  HTTPS_ONLY


  dfs.https.enable
  true





  dfs.namenode.secondary.http-address
  dmp1.example.com:50090<http://dmp1.example.com:50090>


  dfs.secondary.namenode.keytab.file
  /etc/hadoop/conf/hdfs.keytab


  dfs.secondary.namenode.kerberos.principa
  hadoop/_h...@example.com<mailto:h...@example.com>


  dfs.secondary.namenode.kerberos.internal.spnego.principal
  HTTP/_h...@example.com<mailto:h...@example.com>


  dfs.namenode.secondary.https-port
  50470






  dfs.journalnode.keytab.file
  /etc/hadoop/conf/hdfs.keytab


  dfs.journalnode.kerberos.principa
  hadoop/_h...@example.com<mailto:h...@example.com>


  dfs.journalnode.kerberos.internal.spnego.principa
  HTTP/_h...@example.com<mailto:h...@example.com>


  dfs.web.authentication.kerberos.keytab
  /etc/hadoop/conf/hdfs.keytab





  dfs.datanode.kerberos.principal
  hadoop/_h...@example.com<mailto:h...@example.com>


  dfs.datanode.keytab.file
  /etc/hadoop/conf/hdfs.keytab


  dfs.datanode.data.dir.perm
  700




  dfs.datanode.address
  0.0.0.0:61004<http://0.0.0.0:61004>


  dfs.datanode.http.address
  0.0.0.0:61006<http://0.0.0.0:61006>


  dfs.datanode.https.address
  0.0.0.0:50470<http://0.0.0.0:50470>



  dfs.data.transfer.protection
  integrity



 dfs.web.authentication.kerberos.principal
 HTTP/_h...@example.com<mailto:h...@example.com>


 dfs.web.authentication.kerberos.keytab
 /etc/hadoop/conf/hdfs.keytab


and [hadoop@dmp1 hadoop-2.7.3]$ klist -ket /etc/hadoop/conf/hdfs.keytab


Keytab name: FILE:/etc/hadoop/conf/hdfs.keytab
KVNO Timestamp   Principal
 --- --
   2 09/19/2016 16:00:41 
hdfs/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(aes256-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(aes128-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(des3-cbc-sha1)
   2 09/19/2016 16:00:41 
hdfs/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(arcfour-hmac)
   2 09/19/2016 16:00:41 
hdfs/dmp2.example@example.com<mailto:dmp2.example@example.com> 
(aes256-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp2.example@example.com<mailto:dmp2.example@example.com> 
(aes128-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp2.example@example.com<mailto:dmp2.example@example.com> 
(des3-cbc-sha1)
   2 09/19/2016 16:00:41 
hdfs/dmp2.example@example.com<mailto:dmp2.example@example.com> 
(arcfour-hmac)
   2 09/19/2016 16:00:41 
hdfs/dmp3.example@example.com<mailto:dmp3.example@example.com> 
(aes256-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp3.example@example.com<mailto:dmp3.example@example.com> 
(aes128-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
hdfs/dmp3.example@example.com<mailto:dmp3.example@example.com> 
(des3-cbc-sha1)
   2 09/19/2016 16:00:41 
hdfs/dmp3.example@example.com<mailto:dmp3.example@example.com> 
(arcfour-hmac)
   2 09/19/2016 16:00:41 
HTTP/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(aes256-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
HTTP/dmp1.example@example.com<mailto:dmp1.example@example.com> 
(aes128-cts-hmac-sha1-96)
   2 09/19/2016 16:00:41 
HTTP/dmp1.example@example.com<mailto:dmp1.example@example

RE: How to clear block count alert on hdfs

[Brahma Reddy Battula]

Can you elaborate more..?

are asking about why “nondfsused”more..? can you please mention hadoop version 
details also..?



--Brahma Reddy Battula

From: sandeep vura [mailto:sandeepv...@gmail.com]
Sent: 17 September 2016 12:20
To: user@hadoop.apache.org
Subject: How to clear block count alert on hdfs


Hi hadoop experts,

We are getting block count alerts on datanodes. Please find the DFS admin report



Configured Capacity: 58418139463680 (53.13 TB)

Present Capacity: 55931103011017 (50.87 TB)

DFS Remaining: 55237802565632 (50.24 TB)

DFS Used: 693300445385 (645.69 GB)

DFS Used%: 1.24%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

NON DFS USED : 2.26 TB
Another, root voulme is 80%  Utilized on all datanodes.

Kindly please suggest how to clear the block count alert.

RE: Re:

[Brahma Reddy Battula]

Might be it is considered as local filesystem and listing local folders .  It 
can happen with default configuration where “fs.defaultFS” is not configured. 
Please check HADOOP_CONF_DIR is having the hdfs-site.xml and core-site.xml with 
proper configuration.




--Brahma Reddy Battula

From: kishore alajangi [mailto:alajangikish...@gmail.com]
Sent: 19 September 2016 17:19
To: Vinodh Nagaraj
Cc: user-h...@hadoop.apache.org; user@hadoop.apache.org
Subject: Re:

check with -R option.

hadoop fs -ls -R 

On Mon, Sep 19, 2016 at 1:55 PM, Vinodh Nagaraj 
<vinodh.db...@gmail.com<mailto:vinodh.db...@gmail.com>> wrote:
Hi All,

When I execute hdfs dfs -ls,it shows all the directory. I have created one 
directory in hadoop.
Remaining files are created at OS level.

Executing from Hadoop home/bin.


Thanks,




--
Sincere Regards,
A.Kishore Kumar,
Ph: +91 9246274575

RE: [DISCUSS] Retire BKJM from trunk?

[Brahma Reddy Battula]

Yes, We can remove from trunk and  can be deprecated in branch-2. We confirmed 
with all the existing customers on this..


--Brahma Reddy Battula



-Original Message-
From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] 
Sent: 28 July 2016 13:22
To: Rakesh Radhakrishnan; Sijie Guo
Cc: d...@bookkeeper.apache.org; Uma gangumalla; Vinayakumar B; 
hdfs-...@hadoop.apache.org; user@hadoop.apache.org; u...@bookkeeper.apache.org
Subject: Re: [DISCUSS] Retire BKJM from trunk?

For Huawei, Vinay/Brahma should know about their usage. I think after QJM 
stabilized and ready they also adopted to QJM is what I know, but they should 
know more than me as I left that employer while ago.

If no one is using it, It is ok to remove.

Regards,
Uma

On 7/27/16, 9:49 PM, "Rakesh Radhakrishnan" <rake...@apache.org> wrote:

>If I remember correctly, Huawei also adopted QJM component. I hope 
>@Vinay might have discussed internally in Huawei before starting this 
>e-mail discussion thread. I'm +1, for removing the bkjm contrib from 
>the trunk code.
>
>Also, there are quite few open sub-tasks under HDFS-3399 umbrella jira, 
>which was used for the BKJM implementation time. How about closing 
>these jira by marking as "Won't Fix"?
>
>Thanks,
>Rakesh
>Intel
>
>On Thu, Jul 28, 2016 at 1:53 AM, Sijie Guo <si...@apache.org> wrote:
>
>> + Rakesh and Uma
>>
>> Rakesh and Uma might have a better idea on this. I think Huawei was 
>>using  it when Rakesh and Uma worked there.
>>
>> - Sijie
>>
>> On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth 
>><cnaur...@hortonworks.com>
>> wrote:
>>
>> > I recommend including the BookKeeper community in this discussion.
>>I¹ve
>> > added their user@ and dev@ lists to this thread.
>> >
>> > I do not see BKJM being used in practice.  Removing it from trunk
>>would
>> be
>> > attractive in terms of less code for Hadoop to maintain and build,
>>but if
>> > we find existing users that want to keep it, I wouldn¹t object.
>> >
>> > --Chris Nauroth
>> >
>> > On 7/26/16, 11:14 PM, "Vinayakumar B" <vinayakumar...@huawei.com>
>>wrote:
>> >
>> > Hi All,
>> >
>> >BKJM was Active and made much stable when the NameNode HA 
>> > was implemented and there was no QJM implemented.
>> >Now QJM is present and is much stable which is adopted by 
>> > many production environment.
>> >I wonder whether it would be a good time to retire BKJM from
>> trunk?
>> >
>> >Are there any users of BKJM exists?
>> >
>> > -Vinay
>> >
>> >
>> >
>>


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

RE: How to restart an HDFS standby namenode dead for a very long time

[Brahma Reddy Battula]

Sorry for late reply..


To recover this, you can restart the DN’s one by one. (OR) apply the patch in 
HDFS-9917 and then restart the Standby Namenode.


--Brahma Reddy Battula

From: Zach Cox [mailto:zcox...@gmail.com]
Sent: 15 July 2016 19:59
To: Brahma Reddy Battula; user@hadoop.apache.org
Subject: Re: How to restart an HDFS standby namenode dead for a very long time

Yes it's definitely possible we are hitting that jira. Do we need to do 
anything other than rsync dfs.name.dir from the active namenode before starting 
the standby namenode again?

Thanks,
Zach


On Fri, Jul 15, 2016 at 2:21 AM Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:
Seems to be you are hitting following jira.. Please refer

https://issues.apache.org/jira/browse/HDFS-9917




--Brahma Reddy Battula

From: Zach Cox [mailto:zcox...@gmail.com<mailto:zcox...@gmail.com>]
Sent: 14 July 2016 03:34
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: How to restart an HDFS standby namenode dead for a very long time

Hi - we have an HDFS (version 2.0.0-cdh4.4.0) cluster setup in HA with 2 
namenodes and 5 journal nodes. This cluster has been somewhat neglected (long 
story) and the standby namenode process has been dead for several months.

Recently we tried to just start the standby namenode process again, but several 
hours later the entire HDFS cluster (and HBase on top of it) was unavailable 
for several hours. As soon as we stopped the standby namenode process, HDFS 
(and HBase) started working fine again. I don't know for sure, but I'm guessing 
the standby namenode was trying to catch up on several months of edits from 
being down for so long, and just couldn't do it.

We really need to get this standby namenode process started again, so I'm 
trying to find the right way to do it. I've tried starting it with the 
-bootstrapStandby option, but that appears broken in our HDFS version. Instead, 
we can manually rsync the files in the dfs.name.dir from the active namenode.

I guess my question is: is there a recommended way to get this standby namenode 
resurrected successfully? And would we need to do anything other than rsync 
dfs.name.dir from the active namenode before starting the standby namenode 
again?

Thanks,
Zach

RE: How to restart an HDFS standby namenode dead for a very long time

[Brahma Reddy Battula]

Seems to be you are hitting following jira.. Please refer

https://issues.apache.org/jira/browse/HDFS-9917




--Brahma Reddy Battula

From: Zach Cox [mailto:zcox...@gmail.com]
Sent: 14 July 2016 03:34
To: user@hadoop.apache.org
Subject: How to restart an HDFS standby namenode dead for a very long time

Hi - we have an HDFS (version 2.0.0-cdh4.4.0) cluster setup in HA with 2 
namenodes and 5 journal nodes. This cluster has been somewhat neglected (long 
story) and the standby namenode process has been dead for several months.

Recently we tried to just start the standby namenode process again, but several 
hours later the entire HDFS cluster (and HBase on top of it) was unavailable 
for several hours. As soon as we stopped the standby namenode process, HDFS 
(and HBase) started working fine again. I don't know for sure, but I'm guessing 
the standby namenode was trying to catch up on several months of edits from 
being down for so long, and just couldn't do it.

We really need to get this standby namenode process started again, so I'm 
trying to find the right way to do it. I've tried starting it with the 
-bootstrapStandby option, but that appears broken in our HDFS version. Instead, 
we can manually rsync the files in the dfs.name.dir from the active namenode.

I guess my question is: is there a recommended way to get this standby namenode 
resurrected successfully? And would we need to do anything other than rsync 
dfs.name.dir from the active namenode before starting the standby namenode 
again?

Thanks,
Zach

RE: NameNode HA from a client perspective

[Brahma Reddy Battula]

1. Have a list of namenodes, built from configurations.
2. Execute the op on each namenode until its success.
3. Have the successfull namenode url as active namenode, and use the same for 
next operations.
4. Whenever a StandByException or some network exception (other than remote 
exceptions) occurs, then repeat #2 and #3, starting from the next namenode url 
in the list.


--Brahma Reddy Battula

From: Cecile, Adam [mailto:adam.cec...@hitec.lu]
Sent: 04 May 2016 16:26
To: Sandeep Nemuri
Cc: user@hadoop.apache.org
Subject: RE: NameNode HA from a client perspective


Hello,



I'm not sure to understand your answer, may I add a little piece of code:



def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples=[]):

"""

:type hdfs_path: str

:type hdfs_operation: str

"""

if not hdfs_path.startswith("/"):

raise WebHdfsException("The web hdfs path must start with / but 
found " + hdfs_path, None, None)



url = 'http://' + self.host + ':' + str(self.port) + '/webhdfs/v1' + 
hdfs_path + '?user.name=' + self.user + '=' + hdfs_operation

len_param = len(opt_query_param_tuples)

for index in range(len_param):

key_value = opt_query_param_tuples[index]

url += "&{}={}".format(key_value[0], str(key_value[1]))

return url



Here is a plain python standard distribution function extracted from an app: 
the problem here is "self.host", it has to be IP address ou DNS name of the 
NameNode, however I'd like to turn this into something dynamic resolving to the 
current active master.



Regards, Adam.




De : Sandeep Nemuri <nhsande...@gmail.com<mailto:nhsande...@gmail.com>>
Envoyé : mercredi 4 mai 2016 09:15
À : Cecile, Adam
Cc : user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Objet : Re: NameNode HA from a client perspective

I think you can simply use the nameservice (dfs.nameservices) which is defined 
in hdfs-site.xml
The hdfs client should be able to resolve the current active namenode and get 
the necessary information.

Thanks,
Sandeep Nemuri
[https://mailfoogae.appspot.com/t?sender=abmhzYW5kZWVwNkBnbWFpbC5jb20%3D=zerocontent=69e2c096-0009-4482-a881-df6dfb44434f]ᐧ

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam 
<adam.cec...@hitec.lu<mailto:adam.cec...@hitec.lu>> wrote:

Hello All,


I'd like to have a piece of advice regarding how my HDFS clients should handle 
the NameNode high availability feature.
I have a complete setup running with ZKFC and I can see one active and one 
standby NameNode. When I kill the active one, the standy gets active and when 
the original one get back online it turns into a standby node, perfect.

However I'm not sure how my client apps should handle this, a couple of ideas:
* Handle the bad HTTP code from standby node to switch to the other one
* Integrate Zookeeper client to query for the current active node
* Hack something like a shared-ip linked to the active node

Then I'll have to handle a switch that may occurs during the execution of a 
client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.​



--
  Regards
  Sandeep Nemuri

RE: Guideline on setting Namenode RPC Handler count (client and service)

[Brahma Reddy Battula]

Hope you are using hadoop-2.6 release.

As you are targeting to amount of time it’s getting processed, your proposed 
configs options ( ipc.ping.interval and split threshold can be changed)  should 
be fine .  I mean to say, 2nd and 3rd options.

You can try once, let’s know.


Had seen related issue recently , may be you can have look at HDFS-10301.



--Brahma Reddy Battula

From: Chackravarthy Esakkimuthu [mailto:chaku.mi...@gmail.com]
Sent: 03 May 2016 18:10
To: Gokul
Cc: user@hadoop.apache.org
Subject: Re: Guideline on setting Namenode RPC Handler count (client and 
service)

To add more details on why NN startup delayed while setting handler count as 
600.

We are seeing many duplicate full block reports (FBR) from most of the DN's for 
long time (around 3 hours since NN startup) even though NN comes out of safe 
mode in 10 or 15 mins. Since NN comes out of safe mode, duplicate FBR's are not 
rejected.

It's because DN getting timeout (ipc.ping.interval=60s default) on block report 
RPC call before NN completes processing the blockReport RPC call (takes around 
70-80 secs). Hence DN does not realise that FBR got processed and it kept 
trying to send again. But NN has processed it already and gets error only while 
sending output.

The reason why NN takes more than 1 min to process FBR :

  *   FBR contains array of storageBlockReport. (no of data directories 
configured is 10)
  *   Name system write lock is acquired on processing each storageBlockReport 
and hence single handler thread cannot just complete processing FBR completely 
once it acquires the lock.
  *   There is a lock contention with other 599 handler threads who are also 
busy in processing FBR from all DN's. Hence acquiring lock gets delayed and 
then next storageBlockReport gets processed.

 *   t -> storageBlockReport[0]   --> Handler thread starts FBR processing.
 *   t + 5s -> storageBlockReport[1]
 *   t + 12s ->  storageBlockReport[2]
 *   ...
 *   ...
 *   t + 70s -> storageBlockReport[9]  --> Handler thread completes FBR 
processing.

We are looking for some suggestion to resolve this situation of having delayed 
start of NN. (delayed start means even though NN comes out of safe mode, 
because of duplicate FBR, serviceRPC latency remains high and skips the 
heartbeat for more than 1 minute continuously)

Possible config options are :

  1.  Current value for dfs.blockreport.initialDelay is 120s. This can be 
increased to 10 - 15 mins to avoid block report storm.
  2.  Increase ipc.ping.interval from 60s to 90s or so.
  3.  Decrease dfs.blockreport.split.threashold to 100k (from 1M) so that block 
reports from DN will be sent for each storageBlock. Hence DN would get the 
response quickly from NN. But this would delay in sending the heartbeat as each 
RPC call might consume upto 60 secs timeout. Hence heartbeat might get delayed 
for 590s (worst case if all rpc calls succeed consuming 59s).
Or can we move the write lock at higher level and take it once, process all 
storageBlockReports and release it. because from logs, we have seen that each 
storageBlockReport processing takes 20ms-100ms and hence single FBR would 
consume 1s. Also since FBR calls are not that frequent, (block report once in 6 
hours in our cluster / when disk failure happens) Is it ok to reduce the lock 
granularity?

Please give suggestion on the same. Also correct me if I am wrong.

Thanks,
Chackra


On Mon, May 2, 2016 at 2:12 PM, Gokul 
<gokulakanna...@gmail.com<mailto:gokulakanna...@gmail.com>> wrote:
*bump*

On Fri, Apr 29, 2016 at 5:00 PM, Chackravarthy Esakkimuthu 
<chaku.mi...@gmail.com<mailto:chaku.mi...@gmail.com>> wrote:
Hi,

Is there any recommendation or guideline on setting no of RPC handlers in 
Namenode based on cluster size (no of datanodes)?

Cluster details :

No of datanodes - 1200
NN hardware - 74G heap allocated to NN process, 40 core machine
Total blocks - 80M+
Total Files/Directories - 60M+
Total FSObjects - 150M+

We have isolated service and client RPC by enabling service-rpc.

Currently dfs.namenode.handler.count=400 and 
dfs.namenode.service.handler.count=200

Is 200 good fit for this cluster or any change recommended. Please help out.

Thanks in advance!

(We have tried increasing service handler count to 600 and have seen delay in 
NN startup time and then it looked quite stable. And setting it to 200 
decreases the delay in startup time but it has slightly higher rpcQueueTime and 
rpcAvgProcessingTime comparing to 600 handler count.)

Thanks,
Chackra



--
Thanks and Regards,
Gokul

RE: UnstaisfiedLinkError - Windows Environment

[Brahma Reddy Battula]

UnsatisfiedLinkError, is coming because its not able link, Java code API (JNI) 
to corresponding native API in available library.
This will not be the problem from running different version in remote cluster.
Common causes would be:
1. Built for different architecture (32bit?)
2. Libraries corrupted ?

Are you able to read without libraries??
Dont put those libraries in bin and try.

--Brahma Reddy Battula

From: karthi keyan [mailto:karthi93.san...@gmail.com]
Sent: 29 March 2016 16:47
To: Brahma Reddy Battula
Cc: user@hadoop.apache.org
Subject: Re: UnstaisfiedLinkError - Windows Environment

Only side libraries are built with Hadoop 2.6.2 , But Server(Remote Cluster) is 
built with Hadoop 2.5.2? Am not sure that version not be the case here.. Don't 
know wy it happens ? does you knw the cause of this exception ?

RE: UnstaisfiedLinkError - Windows Environment

[Brahma Reddy Battula]

As you told Server is Hadoop 2.5.2.
But client is 2.6.2, as seen in the exception.
Whether client-side libraries built with Hadoop 2.6.2?

From: karthi keyan [mailto:karthi93.san...@gmail.com]
Sent: 29 March 2016 15:16
To: Brahma Reddy Battula
Cc: user@hadoop.apache.org
Subject: Re: UnstaisfiedLinkError - Windows Environment

Yes, built with right libraries.
In my case i have to connect with remote cluster which accommodate Hadoop 
(built for 64 bit windows and Hadoop 2.5.2).

On Tue, Mar 29, 2016 at 12:34 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:
Are you using the right libraries ( built for 64-bit windows and Hadoop 2.6.2) ?

From: karthi keyan 
[mailto:karthi93.san...@gmail.com<mailto:karthi93.san...@gmail.com>]
Sent: 29 March 2016 14:51
To: Brahma Reddy Battula
Cc: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: UnstaisfiedLinkError - Windows Environment

Hi Brahma,

I have added those libraries to the bin path. Every time  when i communicate 
with other cluster(hadoop) am facing this issue.
Is there any Backward compatibility  ?? or some thing else ?

On Tue, Mar 29, 2016 at 12:09 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:
Hadoop Cluster installed in Windows or only client is in Windows?

Whether Hadoop distribution contains windows library files and
/bin is added to PATH ?


From: karthi keyan 
[mailto:karthi93.san...@gmail.com<mailto:karthi93.san...@gmail.com>]
Sent: 29 March 2016 14:29
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: UnstaisfiedLinkError - Windows Environment

Hi,

Frequently am facing this issue while reading the Data from HDFS, Every time i 
have replaced (rebuid) the jars. Does any one suggest me the right way to 
resolve this issue? or can any one tell me the root cause for this error ?

JDK > 1.7
System env - win 64 bit


Caused by: java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V
at 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native Method) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:216)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:807) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:848) 
~[hadoop-hdfs-2.6.2.jar:na]
at java.io.DataInputStream.read(DataInputStream.java:100) 
~[na:1.7.0]

Regards,
Karthikeyan S

RE: UnstaisfiedLinkError - Windows Environment

[Brahma Reddy Battula]

Are you using the right libraries ( built for 64-bit windows and Hadoop 2.6.2) ?

From: karthi keyan [mailto:karthi93.san...@gmail.com]
Sent: 29 March 2016 14:51
To: Brahma Reddy Battula
Cc: user@hadoop.apache.org
Subject: Re: UnstaisfiedLinkError - Windows Environment

Hi Brahma,

I have added those libraries to the bin path. Every time  when i communicate 
with other cluster(hadoop) am facing this issue.
Is there any Backward compatibility  ?? or some thing else ?

On Tue, Mar 29, 2016 at 12:09 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:
Hadoop Cluster installed in Windows or only client is in Windows?

Whether Hadoop distribution contains windows library files and
/bin is added to PATH ?


From: karthi keyan 
[mailto:karthi93.san...@gmail.com<mailto:karthi93.san...@gmail.com>]
Sent: 29 March 2016 14:29
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: UnstaisfiedLinkError - Windows Environment

Hi,

Frequently am facing this issue while reading the Data from HDFS, Every time i 
have replaced (rebuid) the jars. Does any one suggest me the right way to 
resolve this issue? or can any one tell me the root cause for this error ?

JDK > 1.7
System env - win 64 bit


Caused by: java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V
at 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native Method) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:216)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:807) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:848) 
~[hadoop-hdfs-2.6.2.jar:na]
at java.io.DataInputStream.read(DataInputStream.java:100) 
~[na:1.7.0]

Regards,
Karthikeyan S

RE: UnstaisfiedLinkError - Windows Environment

[Brahma Reddy Battula]

Hadoop Cluster installed in Windows or only client is in Windows?

Whether Hadoop distribution contains windows library files and
/bin is added to PATH ?


From: karthi keyan [mailto:karthi93.san...@gmail.com]
Sent: 29 March 2016 14:29
To: user@hadoop.apache.org
Subject: UnstaisfiedLinkError - Windows Environment

Hi,

Frequently am facing this issue while reading the Data from HDFS, Every time i 
have replaced (rebuid) the jars. Does any one suggest me the right way to 
resolve this issue? or can any one tell me the root cause for this error ?

JDK > 1.7
System env - win 64 bit


Caused by: java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V
at 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native Method) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301) 
~[hadoop-common-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:216)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)
 ~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:807) 
~[hadoop-hdfs-2.6.2.jar:na]
at 
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:848) 
~[hadoop-hdfs-2.6.2.jar:na]
at java.io.DataInputStream.read(DataInputStream.java:100) 
~[na:1.7.0]

Regards,
Karthikeyan S

RE: Name node files are creating different directory

[Brahma Reddy Battula]

Hi vinodh

As properties are wrong, it is taken default properties..Correct like 
following..

fs.namenode.name.dir  should be like dfs.namenode.name.dir,
fs.datanode.data.dir  should be like dfs.datanode.data.dir


Reference:
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

From: Vinodh Nagaraj [mailto:vinodh.db...@gmail.com]
Sent: 11 March 2016 13:54
To: user@hadoop.apache.org
Subject: Name node files are creating different directory

Hi,

I configured hadoop 2.7.1 on windows 7 (32 bit ) in C Drive.

I tried to format using "hdfs namenode -format',name node files are created at 
C:\tmp\hadoop-user\dfs\name.but  the property "C:\name" in hdfs-site.xml .

hdfs namenode -format

-10.219.149.100-1457674982841
16/03/11 11:13:03 INFO common.Storage: Storage directory 
\tmp\hadoop-487174\dfs\name has been successfully formatted.

core-site.xml



fs.defaultFS
hdfs://localhost/



hdfs-site.xml



fs.namenode.name.dir
C:\name



fs.datanode.data.dir
C:\data




what is wrong in that?Please help me.

Thanks & Regards,
Vinodh.N

RE: hdfs-site.xml change must restart datanode?

[Brahma Reddy Battula]

Hi yaoxiaohua,
Use "dfs.datanode.max.transfer.threads" instead of "dfs.datanode.max.xcievers 
(deprecated)"..Yes, you need to restart the datanode.
Reference for deprecated 
properties:https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/DeprecatedProperties.html


Thanks And RegardsBrahma Reddy Battula

From: yaoxiao...@outlook.com
To: user@hadoop.apache.org
Subject: hdfs-site.xml change must restart datanode?
Date: Mon, 18 Jan 2016 19:19:24 +0800

Hi,Hadoop2.3cdh5.0.2I change hdfs-site.xml  
  Update two properties value;dfs.datanode.max.xcievers 
dfs.datanode.max.transfer.threads I must 
restart the datanode process for this?Thanks very much! Best 
Regards,Evan Yao

RE: Distcp fails with "Got EOF but currentPos = 240377856 < filelength = 1026034162" error

[Brahma Reddy Battula]

Hi Buntu Dev,

Please check the Data node logs to get the exact root reason.
One more possible reason (apart from kai mentioned)can be direct buffer memory 
is not enough while copying the large files. If you observe the OOM in direct 
buffer, just increase it..

Hope it’s helpful.



From: Buntu Dev [mailto:buntu...@gmail.com]
Sent: 19 January 2016 09:15
To: Zheng, Kai
Cc: user@hadoop.apache.org
Subject: Re: Distcp fails with "Got EOF but currentPos = 240377856 < filelength 
= 1026034162" error

Thanks Kai, but I checked the parqet file that was reported to have issues and 
fsck says the file is healthy.



On Mon, Jan 18, 2016 at 7:09 PM, Zheng, Kai 
> wrote:
Looks like a file it’s copying is ended unexpectedly. Maybe need to find out 
which file, check or read the file in other means to ensure it’s fine not being 
corrupt.

Regards,
Kai

From: Buntu Dev [mailto:buntu...@gmail.com]
Sent: Tuesday, January 19, 2016 5:46 AM
To: user@hadoop.apache.org
Subject: Distcp fails with "Got EOF but currentPos = 240377856 < filelength = 
1026034162" error

I'm using distcp with these options to copy a hdfs directory from one cluster 
to another:


hadoop distcp -prb -i -update -skipcrccheck -delete 
hftp://cluster1/user/hive/warehouse/dir1/ hdfs://cluster2/dir1/


I keep running into these errors related to EOF, what could be causing these 
errors and how to fix this:

~
Caused by: 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: 
java.io.IOException: Got EOF but currentPos = 240377856 < filelength = 
1026034162
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:289)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:257)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)
at 
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
~~


Also I'm using the '-i' to ignore and continue on failures but the distcp does 
retry 3 times and stops. Can anyone throw some light on what else could be 
going wrong.


Thanks!

RE: Max Parallel task executors

[Brahma Reddy Battula]

I'm glad to hear it helped.



Thanks & Regards

 Brahma Reddy Battula





From: sandeep das [yarnhad...@gmail.com]
Sent: Monday, November 09, 2015 11:54 AM
To: user@hadoop.apache.org
Subject: Re: Max Parallel task executors

After increasing yarn.nodemanager.resource.memory-mb to 24 GB more number of 
parallel map tasks are being spawned. Its resolved now.
Thanks a lot for your input.

Regards,
Sandeep

On Mon, Nov 9, 2015 at 9:49 AM, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
BTW Laxman according to the formula that you had provided it turns out that 
only 8 jobs per node will be initiated which is matching with what i'm seeing 
on my setup.

min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb,
 yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcores)


yarn.nodemanager.resource.memory-mb: 16 GB

mapreduce.map.memory.mb: 2 GB

yarn.nodemanager.resource.cpu-vcores: 80

mapreduce.map.cpu.vcores: 1

So if apply the formula then min(16/2, 80/1) -> min(8,80) -> 8

Should i reduce memory per map operation or increase memory for resource 
manager?

On Mon, Nov 9, 2015 at 9:43 AM, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
Thanks Brahma and Laxman for your valuable input.

Following are the statistics available on YARN RM GUI.

Memory Used : 0 GB
Memory Total : 64 GB (16*4 = 64 GB)
VCores Used: 0
VCores Total: 320 (Earlier I had mentioned that I've configured 40 Vcores but 
recently I increased to 80 that's why its appearing 80*4 = 321)

Note: These statistics were captured when there was no job running in 
background.

Let me know whether it was sufficient to nail the issue. If more information is 
required please let me know.

Regards,
Sandeep


On Fri, Nov 6, 2015 at 7:04 PM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote:

The formula for determining the number of concurrently running tasks per node 
is:

min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb,
 yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcores) .


For you scenario :

As you told yarn.nodemanager.resource.memory-mb is configured to 16 GB and 
yarn.nodemanager.resource.cpu-vcores configured to 40. and I am thinking
mapreduce.map/reduce.memory.mb, mapreduce.map/reduce.cpu.vcores default values.

min (16GB/1GB,40Core/1Core )=16 tasks for Node. Then total should be 16*4=64  
(63+1AM)..

I am thinking, Two Nodemanger's are unhealthy (OR) you might have configured 
mapreduce.map/reduce.memory.mb=2GB(or 5 core).

As laxman pointed you can post RMUI or you can cross check like above.

Hope this helps.




Thanks & Regards

 Brahma Reddy Battula





From: Laxman Ch [laxman@gmail.com<mailto:laxman@gmail.com>]
Sent: Friday, November 06, 2015 6:31 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Max Parallel task executors

Can you please copy paste the cluster metrics from RM dashboard.
Its under http://rmhost:port/cluster/cluster

In this page, check under Memory Total vs Memory Used and VCores Total vs 
VCores Used

On 6 November 2015 at 18:21, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
HI Laxman,

Thanks for your response. I had already configured a very high value for 
yarn.nodemanager.resource.cpu-vcores e.g. 40 but still its not increasing more 
number of parallel tasks to execute but if this value is reduced then it runs 
less number of parallel tasks.

As of now yarn.nodemanager.resource.memory-mb is configured to 16 GB and 
yarn.nodemanager.resource.cpu-vcores configured to 40.

Still its not spawning more tasks than 31.

Let me know if more information is required to debug it. I believe there is 
upper limit after which yarn stops spawning tasks. I may be wrong here.


Regards,
Sandeep

On Fri, Nov 6, 2015 at 6:15 PM, Laxman Ch 
<laxman@gmail.com<mailto:laxman@gmail.com>> wrote:
Hi Sandeep,

Please configure the following items to the cores and memory per node you 
wanted to allocate for Yarn containers.
Their defaults are 8 cores and 8GB. So that's the reason you were stuck at 31 
(4nodes * 8cores - 1 AppMaster)

http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
yarn.nodemanager.resource.cpu-vcores
yarn.nodemanager.resource.memory-mb


On 6 November 2015 at 17:59, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
May be to naive to ask but How do I check that?
Sometimes there are almost 200 map tasks pending to run but at a time only 31 
runs.

On Fri, Nov 6, 2015 at 5:57 PM, Chris Mawata 
<chris.maw...@gmail.com<mailto:chris.maw...@gmail.com>> wrote:

Also check that you have more than 31 blocks to process.

On Nov 6, 2015 6:54 AM, "san

RE: unsubscribe

[Brahma Reddy Battula]

Kindly drop a mail to user-unsubscr...@hadoop.apache.org 

Thanks And RegardsBrahma Reddy Battula

Subject: RE: unsubscribe
To: user@hadoop.apache.org
From: wadood.chaudh...@instinet.com
Date: Mon, 9 Nov 2015 11:14:53 -0500

unsubscribe

-
Wadood Chaudhary
309 West 49th Street
New York, NY 10019 US


"Bourre, Marc" ---11/09/2015 11:02:34 AM---unsubscribe

From:"Bourre, Marc" 
To:"user@hadoop.apache.org" , 
Date:11/09/2015 11:02 AM
Subject:RE: unsubscribe




unsubscribe




=




 Disclaimer 


This message is intended solely for use by the named addressee(s). If you 
receive this transmission in error, please immediately notify the sender and 
destroy this message in its entirety, whether in electronic or hard copy 
format. Any unauthorized use (and reliance thereon), copying, disclosure, 
retention, or distribution of this transmission or the material in this 
transmission is forbidden.  We reserve the right to monitor and archive 
electronic communications. This material does not constitute an offer or 
solicitation with respect to the purchase or sale of any security. It should 
not be construed to contain any recommendation regarding any security or 
strategy. Any views expressed are those of the individual sender, except where 
the message states otherwise and the sender is authorized to state them to be 
the views of any such entity. This communication is provided on an “as is” 
basis. It contains material that is owned by Instinet Incorporated, its 
subsidiaries or its or their licensors, and may not, in whole or in part, be 
(i) copied, photocopied or duplicated in any form, by any means, or (ii) 
redistributed, posted, published, excerpted, or quoted without Instinet 
Incorporated's prior written consent. Please access the following link for 
important information and instructions:  
http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt


Securities products and services are provided by locally registered brokerage 
subsidiaries of Instinet Incorporated: Instinet Australia Pty Limited (ACN: 131 
253 686 AFSL No: 327834), regulated by the Australian Securities & Investments 
Commission; Instinet Canada Limited, member IIROC/CIPF; Instinet Pacific 
Limited, authorized and regulated by the Securities and Futures Commission of 
Hong Kong; Instinet Singapore Services Private Limited, regulated by the 
Monetary Authority of Singapore, trading member of The Singapore Exchange 
Securities Trading Private Limited and clearing member of The Central 
Depository (Pte) Limited; and Instinet, LLC, member SIPC.





=

  

RE: Max Parallel task executors

[Brahma Reddy Battula]

The formula for determining the number of concurrently running tasks per node 
is:

min (yarn.nodemanager.resource.memory-mb / mapreduce.[map|reduce].memory.mb,
 yarn.nodemanager.resource.cpu-vcores / mapreduce.[map|reduce].cpu.vcores) .


For you scenario :

As you told yarn.nodemanager.resource.memory-mb is configured to 16 GB and 
yarn.nodemanager.resource.cpu-vcores configured to 40. and I am thinking
mapreduce.map/reduce.memory.mb, mapreduce.map/reduce.cpu.vcores default values.

min (16GB/1GB,40Core/1Core )=16 tasks for Node. Then total should be 16*4=64  
(63+1AM)..

I am thinking, Two Nodemanger's are unhealthy (OR) you might have configured 
mapreduce.map/reduce.memory.mb=2GB(or 5 core).

As laxman pointed you can post RMUI or you can cross check like above.

Hope this helps.




Thanks & Regards

 Brahma Reddy Battula





From: Laxman Ch [laxman@gmail.com]
Sent: Friday, November 06, 2015 6:31 PM
To: user@hadoop.apache.org
Subject: Re: Max Parallel task executors

Can you please copy paste the cluster metrics from RM dashboard.
Its under http://rmhost:port/cluster/cluster

In this page, check under Memory Total vs Memory Used and VCores Total vs 
VCores Used

On 6 November 2015 at 18:21, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
HI Laxman,

Thanks for your response. I had already configured a very high value for 
yarn.nodemanager.resource.cpu-vcores e.g. 40 but still its not increasing more 
number of parallel tasks to execute but if this value is reduced then it runs 
less number of parallel tasks.

As of now yarn.nodemanager.resource.memory-mb is configured to 16 GB and 
yarn.nodemanager.resource.cpu-vcores configured to 40.

Still its not spawning more tasks than 31.

Let me know if more information is required to debug it. I believe there is 
upper limit after which yarn stops spawning tasks. I may be wrong here.


Regards,
Sandeep

On Fri, Nov 6, 2015 at 6:15 PM, Laxman Ch 
<laxman@gmail.com<mailto:laxman@gmail.com>> wrote:
Hi Sandeep,

Please configure the following items to the cores and memory per node you 
wanted to allocate for Yarn containers.
Their defaults are 8 cores and 8GB. So that's the reason you were stuck at 31 
(4nodes * 8cores - 1 AppMaster)

http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
yarn.nodemanager.resource.cpu-vcores
yarn.nodemanager.resource.memory-mb


On 6 November 2015 at 17:59, sandeep das 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
May be to naive to ask but How do I check that?
Sometimes there are almost 200 map tasks pending to run but at a time only 31 
runs.

On Fri, Nov 6, 2015 at 5:57 PM, Chris Mawata 
<chris.maw...@gmail.com<mailto:chris.maw...@gmail.com>> wrote:

Also check that you have more than 31 blocks to process.

On Nov 6, 2015 6:54 AM, "sandeep das" 
<yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>> wrote:
Hi Varun,

I tried to increase this parameter but it did not increase number of parallel 
tasks but if It is decreased then YARN reduces number of parallel tasks. I'm 
bit puzzled why its not increasing more than 31 tasks even after its value is 
increased.

Is there any other configuration as well which controls on how many maximum 
tasks can execute in parallel?

Regards,
Sandeep

On Tue, Nov 3, 2015 at 7:29 PM, Varun Vasudev 
<vvasu...@apache.org<mailto:vvasu...@apache.org>> wrote:
The number of parallel tasks that are run depends on the amount of memory and 
vcores on your machines and the amount of memory and vcores required by your 
mappers and reducers. The amount of memory can be set via 
yarn.nodemanager.resource.memory-mb(the default is 8G). The amount of vcores 
can be set via yarn.nodemanager.resource.cpu-vcores(the default is 8 vcores).

-Varun

From: sandeep das <yarnhad...@gmail.com<mailto:yarnhad...@gmail.com>>
Reply-To: <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Monday, November 2, 2015 at 3:56 PM
To: <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Max Parallel task executors

Hi Team,

I've a cloudera cluster of 4 nodes. Whenever i submit a job my only 31 parallel 
tasks are executed whereas my machines have more CPU available but still 
YARN/AM does not create more task.

Is there any configuration which I can change to start more MAP/REDUCER task in 
parallel?

Each machine in my cluster has 24 CPUs.

Regards,
Sandeep





--
Thanks,
Laxman




--
Thanks,
Laxman

RE: Unsubscribe footer for user@h.a.o messages

[Brahma Reddy Battula]

+ 1 ( non-binding)..
Nice thought,Arpit..


Thanks And RegardsBrahma Reddy Battula

Subject: Re: Unsubscribe footer for user@h.a.o messages
From: m...@hortonworks.com
To: user@hadoop.apache.org
Date: Thu, 5 Nov 2015 21:23:41 +






+1 (non-binding)











On Nov 5, 2015, at 12:50 PM, Arpit Agarwal  wrote:






Apache project mailing lists can add unsubscribe footers to messages. E.g. 
>From spark-user.
https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3C5637830F.3070702%40uib.no%3E



If no one objects I will file an INFRA ticket to add the footer to 
user@h.a.o. Unsubscribe requests are less frequent on the dev mailing lists so 
we can leave those alone.












  

RE: hadoop not using whole disk for HDFS

[Brahma Reddy Battula]

For each configured dfs.datanode.data.dir , HDFS thinks its in separate 
partiotion and counts the capacity separately. So when another dir is added 
/hdfs/data, HDFS thinks new partition is added, So it increased the capacity 
50GB per node. i.e. 100GB for 2 Nodes.

Not allowing /home directory to configure for data.dir might be ambari's 
constraint, instead you can manually try to add a data dir in /home, for your 
usecase, and restart datanodes.




Thanks & Regards

 Brahma Reddy Battula





From: Naganarasimha G R (Naga) [garlanaganarasi...@huawei.com]
Sent: Friday, November 06, 2015 7:20 AM
To: user@hadoop.apache.org
Subject: RE: hadoop not using whole disk for HDFS


Hi Bob,



1. I wasn’t able to set the config to /home/hdfs/data. I got an error that told 
me I’m not allowed to set that config to the /home directory. So I made it 
/hdfs/data.

Naga : I am not sure about the HDP Distro but if you make it point to 
/hdfs/data, still it will be pointing to the root mount itself i.e.

/dev/mapper/centos-root 50G 12G 39G 23% /

Other Alternative is to mount the drive to some other folder other than /home 
and then try.



2. When I restarted, the space available increased by a whopping 100GB.

Naga : I am particularly not sure how this happened may be you can again 
recheck if you enter the command "df -h "  
you will find out how much disk space is available on the related mount for 
which the path is configured.



Regards,

+ Naga









From: Adaryl "Bob" Wakefield, MBA [adaryl.wakefi...@hotmail.com]
Sent: Friday, November 06, 2015 06:54
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

Is there a maximum amount of disk space that HDFS will use? Is 100GB that max? 
When we’re supposed to be dealing with “big data” why is the amount of data to 
be held on any one box such a small number when you’ve got terabytes available?

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Adaryl "Bob" Wakefield, MBA<mailto:adaryl.wakefi...@hotmail.com>
Sent: Wednesday, November 04, 2015 4:38 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: hadoop not using whole disk for HDFS

This is an experimental cluster and there isn’t anything I can’t lose. I ran 
into some issues. I’m running the Hortonworks distro and am managing things 
through Ambari.

1. I wasn’t able to set the config to /home/hdfs/data. I got an error that told 
me I’m not allowed to set that config to the /home directory. So I made it 
/hdfs/data.
2. When I restarted, the space available increased by a whopping 100GB.



Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Naganarasimha G R (Naga)<mailto:garlanaganarasi...@huawei.com>
Sent: Wednesday, November 04, 2015 4:26 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: hadoop not using whole disk for HDFS


Better would be to stop the daemons and copy the data from /hadoop/hdfs/data to 
/home/hdfs/data , reconfigure dfs.datanode.data.dir to /home/hdfs/data and then 
start the daemons. If the data is comparitively less !

Ensure you have the backup if have any critical data !



Regards,

+ Naga


From: Adaryl "Bob" Wakefield, MBA [adaryl.wakefi...@hotmail.com]
Sent: Thursday, November 05, 2015 03:40
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

So like I can just create a new folder in the home directory like:
home/hdfs/data
and then set dfs.datanode.data.dir to:
/hadoop/hdfs/data,home/hdfs/data

Restart the node and that should do it correct?

Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData

From: Naganarasimha G R (Naga)<mailto:garlanaganarasi...@huawei.com>
Sent: Wednesday, November 04, 2015 3:59 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: hadoop not using whole disk for HDFS


Hi Bob,



Seems like you have configured to disk dir to be other than an folder in /home, 
if so try creating another folder and add to "dfs.datanode.data.dir" seperated 
by comma instead of trying to reset the default.

And its also advised not to use the root partition "/" to be configured for 
HDFS data dir, if the Dir usage hits the maximum then OS might fail to function 
properly.



Regards,

+ Naga


From: P lva [ruvi...@gmail.com]
Sent: Thursday, November 05, 2015 03:11
To: user@hadoop.apache.org
Subject: Re: hadoop not using whole disk for HDFS

What does your dfs.datanode.data.dir point to ?


On Wed, Nov 4, 2015 at 4:1

RE: Problem running example (wrong IP address)

[Brahma Reddy Battula]

Thanks for sharing the logs.
Problem is interesting..can you please post namenode logs and dual IP 
configurations(thinking problem with gateway while sending requests from 52.1 
segment to 51.1 segment..)

Thanks And RegardsBrahma Reddy Battula

Date: Fri, 25 Sep 2015 12:19:00 -0500
Subject: Re: Problem running example (wrong IP address)
From: dwmaill...@gmail.com
To: user@hadoop.apache.org

hadoop-master http://pastebin.com/yVF8vCYShadoop-data1 
http://pastebin.com/xMEdf01ehadoop-data2 http://pastebin.com/prqd02eZ


On Fri, Sep 25, 2015 at 11:53 AM, Brahma Reddy Battula 
<brahmareddy.batt...@hotmail.com> wrote:



sorry,I am not able to access the logs, could please post in paste bin or 
attach the 192.168.51.6( as your query is why different IP) DN logs and 
namenode logs here..?



Thanks And RegardsBrahma Reddy Battula

Date: Fri, 25 Sep 2015 11:16:55 -0500
Subject: Re: Problem running example (wrong IP address)
From: dwmaill...@gmail.com
To: user@hadoop.apache.org

Brahma,
Thanks for the reply. I'll keep this conversation here in the user list. The 
/etc/hosts file is identical on all three nodes
hadoop@hadoop-data1:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
hadoop@hadoop-data2:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
hadoop@hadoop-master:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
Here are the startup logs for all three 
nodes:https://gist.github.com/dwatrous/7241bb804a9be8f9303f
https://gist.github.com/dwatrous/bcd85cda23d6eca3a68b
https://gist.github.com/dwatrous/922c4f773aded0137fa3

Thanks for your help.

On Fri, Sep 25, 2015 at 10:33 AM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com> wrote:







Seems DN started in three machines and failed in hadoop-data1(192.168.52.4)..





192.168.51.6 : giving IP as 192.168.51.1...can you please check /etc/hosts file 
of 192.168.51.6 (might be 192.168.51.1 is configured in /etc/hosts)



192.168.52.4 : datanode startup might be failed ( you can check this node logs)



192.168.51.4 :  Datanode starup is success..which is in master node..









Thanks & Regards

 Brahma Reddy Battula

 










From: Daniel Watrous [dwmaill...@gmail.com]

Sent: Friday, September 25, 2015 8:41 PM

To: user@hadoop.apache.org

Subject: Re: Problem running example (wrong IP address)






I'm still stuck on this and posted it to stackoverflow:
http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address





Thanks,
Daniel



On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


I could really use some help here. As you can see from the output below, the 
two attached datanodes are identified with a non-existent IP address. Can 
someone tell me how that gets selected or how to explicitly set it. Also, why 
are both datanodes
 shown under the same name/IP?





hadoop@hadoop-master:~$ hdfs dfsadmin -report
Configured Capacity: 84482326528 (78.68 GB)
Present Capacity: 75745546240 (70.54 GB)
DFS Remaining: 75744862208 (70.54 GB)
DFS Used: 684032 (668 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0



-
Live datanodes (2):



Name: 192.168.51.1:50010 (192.168.51.1)
Hostname: hadoop-data1
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 303104 (296 KB)
Non DFS Used: 
4302479360 (4.01 GB)
DFS Remaining: 37938380800 (35.33 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.81%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:37 UTC 2015






Name: 192.168.51.4:50010 (hadoop-master)
Hostname: hadoop-master
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 380928 (372 KB)
Non DFS Used: 
4434300928 (4.13 GB)
DFS Remaining: 37806481408 (35.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.50%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:38 UTC 2015













On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


The IP address is clearly wrong, but I'm not sure how it gets set. Can someone 
tell me how to configure it to choose a valid IP address?




On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


I just noticed that both datanodes appear to have chosen that IP address and 
bound that port for HDFS communication.



http://screencast.com/t/OQNbrWFF





Any idea why this would be? Is there some way to specify which IP/hostname 
should be used for that?





On Thu, Sep 24, 2015 at

RE: Problem running example (wrong IP address)

[Brahma Reddy Battula]

Seems DN started in three machines and failed in hadoop-data1(192.168.52.4)..


192.168.51.6 : giving IP as 192.168.51.1<http://192.168.51.1:50010>...can you 
please check /etc/hosts file of 192.168.51.6 (might be 
192.168.51.1<http://192.168.51.1:50010> is configured in /etc/hosts)

192.168.52.4 : datanode startup might be failed ( you can check this node logs)

192.168.51.4 : <http://192.168.51.4:50010>  Datanode starup is success..which 
is in master node..




Thanks & Regards

 Brahma Reddy Battula





From: Daniel Watrous [dwmaill...@gmail.com]
Sent: Friday, September 25, 2015 8:41 PM
To: user@hadoop.apache.org
Subject: Re: Problem running example (wrong IP address)

I'm still stuck on this and posted it to stackoverflow:
http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address

Thanks,
Daniel

On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous 
<dwmaill...@gmail.com<mailto:dwmaill...@gmail.com>> wrote:
I could really use some help here. As you can see from the output below, the 
two attached datanodes are identified with a non-existent IP address. Can 
someone tell me how that gets selected or how to explicitly set it. Also, why 
are both datanodes shown under the same name/IP?

hadoop@hadoop-master:~$ hdfs dfsadmin -report
Configured Capacity: 84482326528 (78.68 GB)
Present Capacity: 75745546240 (70.54 GB)
DFS Remaining: 75744862208 (70.54 GB)
DFS Used: 684032 (668 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-
Live datanodes (2):

Name: 192.168.51.1:50010<http://192.168.51.1:50010> (192.168.51.1)
Hostname: hadoop-data1
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 303104 (296 KB)
Non DFS Used: 4302479360 (4.01 GB)
DFS Remaining: 37938380800 (35.33 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.81%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:37 UTC 2015


Name: 192.168.51.4:50010<http://192.168.51.4:50010> (hadoop-master)
Hostname: hadoop-master
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 380928 (372 KB)
Non DFS Used: 4434300928 (4.13 GB)
DFS Remaining: 37806481408 (35.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.50%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:38 UTC 2015



On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous 
<dwmaill...@gmail.com<mailto:dwmaill...@gmail.com>> wrote:
The IP address is clearly wrong, but I'm not sure how it gets set. Can someone 
tell me how to configure it to choose a valid IP address?

On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous 
<dwmaill...@gmail.com<mailto:dwmaill...@gmail.com>> wrote:
I just noticed that both datanodes appear to have chosen that IP address and 
bound that port for HDFS communication.

http://screencast.com/t/OQNbrWFF

Any idea why this would be? Is there some way to specify which IP/hostname 
should be used for that?

On Thu, Sep 24, 2015 at 3:11 PM, Daniel Watrous 
<dwmaill...@gmail.com<mailto:dwmaill...@gmail.com>> wrote:
When I try to run a map reduce example, I get the following error:

hadoop@hadoop-master:~$ hadoop jar 
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 
10 30
Number of Maps  = 10
Samples per Map = 30
15/09/24 20:04:28 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Got error, status message , ack with firstBadLink as 
192.168.51.1:50010<http://192.168.51.1:50010>
at 
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1334)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
15/09/24 20:04:28 INFO hdfs.DFSClient: Abandoning 
BP-852923283-127.0.1.1-1443119668806:blk_1073741825_1001
15/09/24 20:04:28 INFO hdfs.DFSClient: Excluding datanode 
DatanodeInfoWithStorage[192.168.51.1:50010<http://192.168.51.1:50010>,DS-45f6e06d-752e-41e8-ac25-ca88bce80d00,DISK]
15/09/24 20:04:28 WARN hdfs.DFSClient: Slow waitForAckedSeqno took 65357ms 
(threshold=3ms)
Wrote input for Map #0

I'm not sure why it's trying to access 
192.168.51.1:50010<http://192.168.51.1:50010>, which isn't even a valid IP 
address in my setup.

Daniel

RE: Problem running example (wrong IP address)

[Brahma Reddy Battula]

sorry,I am not able to access the logs, could please post in paste bin or 
attach the 192.168.51.6( as your query is why different IP) DN logs and 
namenode logs here..?



Thanks And RegardsBrahma Reddy Battula

Date: Fri, 25 Sep 2015 11:16:55 -0500
Subject: Re: Problem running example (wrong IP address)
From: dwmaill...@gmail.com
To: user@hadoop.apache.org

Brahma,
Thanks for the reply. I'll keep this conversation here in the user list. The 
/etc/hosts file is identical on all three nodes
hadoop@hadoop-data1:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
hadoop@hadoop-data2:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
hadoop@hadoop-master:~$ cat /etc/hosts127.0.0.1 localhost192.168.51.4 
hadoop-master
192.168.52.4 hadoop-data1192.168.52.6 hadoop-data2
Here are the startup logs for all three 
nodes:https://gist.github.com/dwatrous/7241bb804a9be8f9303f
https://gist.github.com/dwatrous/bcd85cda23d6eca3a68b
https://gist.github.com/dwatrous/922c4f773aded0137fa3

Thanks for your help.

On Fri, Sep 25, 2015 at 10:33 AM, Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com> wrote:







Seems DN started in three machines and failed in hadoop-data1(192.168.52.4)..





192.168.51.6 : giving IP as 192.168.51.1...can you please check /etc/hosts file 
of 192.168.51.6 (might be 192.168.51.1 is configured in /etc/hosts)



192.168.52.4 : datanode startup might be failed ( you can check this node logs)



192.168.51.4 :  Datanode starup is success..which is in master node..









Thanks & Regards

 Brahma Reddy Battula

 










From: Daniel Watrous [dwmaill...@gmail.com]

Sent: Friday, September 25, 2015 8:41 PM

To: user@hadoop.apache.org

Subject: Re: Problem running example (wrong IP address)






I'm still stuck on this and posted it to stackoverflow:
http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address





Thanks,
Daniel



On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


I could really use some help here. As you can see from the output below, the 
two attached datanodes are identified with a non-existent IP address. Can 
someone tell me how that gets selected or how to explicitly set it. Also, why 
are both datanodes
 shown under the same name/IP?





hadoop@hadoop-master:~$ hdfs dfsadmin -report
Configured Capacity: 84482326528 (78.68 GB)
Present Capacity: 75745546240 (70.54 GB)
DFS Remaining: 75744862208 (70.54 GB)
DFS Used: 684032 (668 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0



-
Live datanodes (2):



Name: 192.168.51.1:50010 (192.168.51.1)
Hostname: hadoop-data1
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 303104 (296 KB)
Non DFS Used: 
4302479360 (4.01 GB)
DFS Remaining: 37938380800 (35.33 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.81%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:37 UTC 2015






Name: 192.168.51.4:50010 (hadoop-master)
Hostname: hadoop-master
Decommission Status : Normal
Configured Capacity: 42241163264 (39.34 GB)
DFS Used: 380928 (372 KB)
Non DFS Used: 
4434300928 (4.13 GB)
DFS Remaining: 37806481408 (35.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 89.50%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri Sep 25 13:25:38 UTC 2015













On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


The IP address is clearly wrong, but I'm not sure how it gets set. Can someone 
tell me how to configure it to choose a valid IP address?




On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:


I just noticed that both datanodes appear to have chosen that IP address and 
bound that port for HDFS communication.



http://screencast.com/t/OQNbrWFF





Any idea why this would be? Is there some way to specify which IP/hostname 
should be used for that?





On Thu, Sep 24, 2015 at 3:11 PM, Daniel Watrous 
<dwmaill...@gmail.com> wrote:



When I try to run a map reduce example, I get the following error:




hadoop@hadoop-master:~$ hadoop jar 
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 
10 30
Number of Maps  = 10
Samples per Map = 30
15/09/24 20:04:28 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Got error, status message , ack with firstBadLink as 
192.168.51.1:50010
at 
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
at 
org.

RE: unsubscribe

[Brahma Reddy Battula]

Kindly Drop a mail to user-unsubscr...@hadoop.apache.org



Thanks & Regards

 Brahma Reddy Battula





From: Jean-Eric CAZAMEA [jecaza...@yahoo.com]
Sent: Friday, September 18, 2015 1:17 AM
To: User Hadoop
Subject: unsubscribe

RE: unsubscribe me

[Brahma Reddy Battula]

Drop a mail to user-unsubscr...@hadoop.apache.org


Thanks And RegardsBrahma Reddy Battula

Date: Wed, 16 Sep 2015 19:48:26 +0530
Subject: unsubscribe me
From: akfak...@gmail.com
To: user@hadoop.apache.org


  

RE: unsubscribe

[Brahma Reddy Battula]

Kindly Drop a mail to user-unsubscr...@hadoop.apache.org


Thanks & Regards

 Brahma Reddy Battula





From: dhvanika.s...@wipro.com [dhvanika.s...@wipro.com]
Sent: Monday, September 14, 2015 10:09 AM
To: user@hadoop.apache.org
Subject: unsubscribe


The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

RE: Re:RE: Unable to use ./hdfs dfsadmin -report with HDFS Federation

[Brahma Reddy Battula]

To get dfsadmin report in the federated cluster, you need to do configure 
core-site.xml like below.( client side)

1) Configure all the nameservices
2) Configure the fs.defaultFS to corresponding name service..

Example:

If we Four name services are running in federated cluster, you can configure 
like following..

property
namefs.viewfs.mounttable.ClusterX.link./dir2/name
valuehdfs://ns1/dir1/value
/property

property
namefs.viewfs.mounttable.ClusterX.link./dir3/name
valuehdfs://ns2/dir2/value
/property

property
namefs.viewfs.mounttable.ClusterX.link./dir4/name
valuehdfs://ns3/dir3/value
/property

property
namefs.viewfs.mounttable.ClusterX.link./dir5/name
valuehdfs://ns4/dir4/value
/property

And if you want get admin report for ns1 and configure like below.

property
namefs.defaultFS/name
valuehdfs://ns1/value
/property

Hence it will print details for ns1..



For more details check following.

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/ViewFs.html


Hope this can answer your query.




Thanks  Regards

Brahma Reddy Battula





From: Todd [bit1...@163.com]
Sent: Monday, August 24, 2015 9:33 AM
To: user@hadoop.apache.org
Subject: Re:RE: Unable to use ./hdfs dfsadmin -report with HDFS Federation


Thanks Chinnappan for the quick reply!
 Looks that the command hdfs dfsadmin -report should work if I want to see the 
whole cluster's status. But as you see, I see the message that view.fs is not   
HDFS file system.
If I want to see part of the whole cluster, then how I could go on?
could you please show me the command? Thanks!




At 2015-08-24 11:57:26, Chinnappan Chandrasekaran chiranchan...@jos.com.sg 
wrote:

hdfs dfsadmin –report

The above command will show entire cluster status

From: Todd [mailto:bit1...@163.commailto:bit1...@163.com]
Sent: Monday, 24 August, 2015 11:54 AM
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
Subject: Unable to use ./hdfs dfsadmin -report with HDFS Federation

Hi, our hadoop cluster is using HDFS Federation, but when use the following 
command to report the HDFS status
It gives me the following message that viewfs is NOT HDFS filesystem
Then how can I proceed to report the hdfs status
Thanks!

[admin@namenode1]$ ./hdfs dfsadmin -report
report: FileSystem viewfs://nsX/ is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning]

__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__
[http://img.photobucket.com/albums/v232/RiZZo66/WLAchieverAward2014BampW.jpg]
For general enquiries, please contact us at JOS Enquiry Email: 
enqu...@jos.com.sgmailto:enqu...@jos.com.sg Hotline: (+65) 6551 9611
For JOS Support, please contact us at JOS Services Email: 
servi...@jos.com.sgmailto:servi...@jos.com.sg Hotline: (+65) 6484 2302

A member of the Jardine Matheson Group, Jardine OneSolution is one of Asia’s 
leading providers of integrated IT services and solutions with offices in 
Singapore, Malaysia, Hong Kong and China. Find out more about JOS at 
www.jos.comhttp://www.jos.com

Confidentiality Notice and Disclaimer:
This email (including any attachment to it) is confidential and intended only 
for the use of the individual or entity named above and may contain information 
that is privileged. If you are not the intended recipient, you are notified 
that any dissemination, distribution or copying of this email is strictly 
prohibited. If you have received this email in error, please notify us 
immediately by return email or telephone and destroy the original message 
(including any attachment to it). Thank you.



__
This email has been scanned by the Symantec Email Security.cloud service.
Confidentiality Notice and Disclaimer: This email (including any attachment to 
it) is confidential and intended only for the use of the individual or entity 
named above and may contain information that is privileged. If you are not the 
intended recipient, you are notified that any dissemination, distribution or 
copying of this email is strictly prohibited. If you have received this email 
in error, please notify us immediately by return email or telephone and destroy 
the original message (including any attachment to it). Thank you.
__

RE: NNBench on external HDFS

[Brahma Reddy Battula]

HI Alex

HDFS-7651 is raised for same..Please have a look at once..

Thanks  Regards
 Brahma Reddy Battula
___
From: Alexander Striffeler [a.striffe...@students.unibe.ch]
Sent: Thursday, July 23, 2015 1:00 PM
To: user@hadoop.apache.org
Subject: Re: NNBench on external HDFS

Hi Chris,

Wow, thanks a lot for your swift and extensive response! I'll try your
suggestion with the local copy and in a second step I'll open a jira
request...

Have a good day,
--Alex

On 22.07.2015 20:14, Chris Nauroth wrote:
 Hi Alexander,

 Your NNBench usage looks basically correct, but NNBench is not a standard
 Hadoop tool.  It does not implement the org.apache.hadoop.util.Tool
 interface, it does not execute through org.apache.hadoop.util.ToolRunner,
 and therefore it does not support the command line arguments that a lot of
 other Hadoop tools like the FsShell support.  Specifically, it does not
 support passing -D arguments to override fs.defaultFS or any other
 configuration properties.

 An alternative way to handle this would be to get a local copy of the
 configuration directory from the remote cluster that you want to test.  I
 expect those configuration files would have fs.defaultFS set to the URL of
 that remote cluster in core-site.xml.  Before launching NNBench, run
 export HADOOP_CONF_DIR=path to local copy of configuration files.
 After exporting that environment variable, you can run hadoop classpath
 to print the classpath that will be used by all hadoop commands and
 confirm that the correct configuration directory for the target cluster is
 on the classpath.  Then, you can run NNBench again, but drop the -D
 argument, since it's going to get ignored anyway.

 I don't see any reason why NNBench shouldn't implement the standard Tool
 interface and thus support the command line arguments that you were
 expecting.  If you'd like to request that as an enhancement, please go
 ahead and file an HDFS jira to request it.  Feel free to post a patch too
 if you're inclined.  Otherwise, someone else in the community can pick it
 up.

 I hope this helps.

 --Chris Nauroth




 On 7/22/15, 12:41 AM, Alexander Striffeler
 a.striffe...@students.unibe.ch wrote:

 Hi all

 I'm pretty new to the Hadoop environment and I'm about performing some
 micro benchmarks. In particular, I'm struggling with executing NNBench
 against an external File System:

 hadoop jar
 /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t
 ests.jar
 nnbench -Dfs.defaultFS='hfds://external.file.system' -operation
 create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100
 -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s`

 yields in
 java.lang.IllegalArgumentException: Wrong FS:
 hdfs://external.file.system/user/username/nnbench-hostname/data,
 expected: hdfs://native fs

 If I neglect the ext FS prefix in the baseDir, NNBench simply ignores
 the -D option and writes the files to the native DFS. Does anyone have
 an idea how to solve this and nnbench an external DFS?

 Thanks a lot, any hints are very appreciated!
 Regards,
 Alex

RE: Incorrect configuration issue

[Brahma Reddy Battula]

you need to configure resolved host name (hdfs://kyahadmaster:54310)

configure like following and start cluster.


configuration
property
namefs.defaultFS/name
valuehdfs://localhost:9000/value
/property
/configuration


For more details, check the following link

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html






Thanks  Regards

 Brahma Reddy Battula





From: khalid yatim [yatimkha...@gmail.com]
Sent: Monday, July 13, 2015 11:16 PM
To: user@hadoop.apache.org
Subject: Re: Incorrect configuration issue


Hello,

I'm expressing some difficulties making a single node hadoop (2.6.0) install 
working.

My confs files seem to be OK. but I'm getting this errors

Incorrect configuration: namenode address dfs.namenode.servicerpc-address or 
dfs.namenode.rpc-address is not configured.

In hadoop-user-namenode-machine.log, I'm getting
Invalid URI for NameNode address (check fs.defaultFS): file:/// has no 
authority.

Here is the content of my core-site.xml file

?xml version=1.0 encoding=UTF-8?
?xml-stylesheet type=text/xsl href=configuration.xsl?
!--
  Licensed under the Apache License, Version 2.0 (the License);
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an AS IS BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
--

!-- Put site-specific property overrides in this file. --

configuration
property
namehadoop.tmp.dir/name
value/app/hadoop/app/value
descriptionTemporary Directory./description
/property
property
namefs.default.namehttp://fs.default.name/name
valuehdfs://kyahadmaster:54310/value
descriptionUse HDFS as file storage engine/description
/property
/configuration


2015-07-13 17:26 GMT+00:00 khalid yatim 
yatimkha...@gmail.commailto:yatimkha...@gmail.com:
Hello,

I'm expressing some difficulties making a single node hadoop (2.6.0) install 
working.

My confs files seem to be OK. but I'm getting this errors

Incorrect configuration: namenode address dfs.namenode.servicerpc-address or 
dfs.namenode.rpc-address is not configured.

In hadoop-user-namenode-machine.log, I'm getting
Invalid URI for NameNode address (check fs.defaultFS): file:/// has no 
authority.


how can I configure logs to get more explicit information about what's going 
wrong?


I'm new here!

Thank you.

--
YATIM Khalid
06 76 19 87 95

INGENIEUR ENSIASTE
Promotion 2007



--
YATIM Khalid
06 76 19 87 95

INGENIEUR ENSIASTE
Promotion 2007

RE: Cannot obtain block length for LocatedBlock

[Brahma Reddy Battula]

Can you try like following..?

hdfs fsck -openforwrite -files -blocks -locations / | grep 
blk_1109280129_1099547327549



Thanks  Regards

 Brahma Reddy Battula



From: Adnan Karač [adnanka...@gmail.com]
Sent: Tuesday, May 26, 2015 1:34 PM
To: user@hadoop.apache.org
Subject: Cannot obtain block length for LocatedBlock

Hi all,

I have an MR job running and exiting with following exception.

java.io.IOException: Cannot obtain block length for LocatedBlock
{BP-1632531813-172.19.67.67-1393407344218:blk_1109280129_1099547327549; 
getBlockSize()=139397; corrupt=false; offset=0; 
locs=[172.19.67.67:50010http://172.19.67.67:50010, 
172.19.67.78:50010http://172.19.67.78:50010, 
172.19.67.84:50010http://172.19.67.84:50010]}

Now, the fun part is that i don't know which file is in question. In order to 
find this out, i did this:

hdfs fsck -files -blocks  / | grep blk_1109280129_1099547327549

Interestingly enough, it came up with nothing.

Did anyone experience anything similar? Or does anyone have a piece of advice 
on how to resolve this?

Version of hadoop is 2.3.0

Thanks in advance!

--
Adnan Karač
[https://mailfoogae.appspot.com/t?sender=aYWRuYW5rYXJhY0BnbWFpbC5jb20%3Dtype=zerocontentguid=316827dc-8cb2-45d7-a776-5c8b1d11bc17]ᐧ

RE: Please unsubscribe Me.

[Brahma Reddy Battula]

Please send email to user-unsubscr...@hadoop.apache.orghttp://apache.org




Thanks  Regards

 Brahma Reddy Battula





From: Aqeel Ahmed [aqeel@gmail.com]
Sent: Tuesday, May 26, 2015 6:47 PM
To: user@hadoop.apache.org
Subject: Please unsubscribe Me.

Please Unsubscribe Me.


aqeel@gmail.commailto:aqeel@gmail.com

--

Regards,

Aqeel Ahmed

RE: unsubscribe

[Brahma Reddy Battula]

Please send email to user-unsubscr...@hadoop.apache.org

Date: Thu, 23 Apr 2015 09:23:37 -0700
Subject: unsubscribe
From: nelsonq4t...@gmail.com
To: user@hadoop.apache.org; nandakumar.vadiv...@ericsson.com

unsubscribe   

RE: Unable to load native-hadoop library

[Brahma Reddy Battula]

Hello Mich Talebzadeh
Please mention which release you are using and how did you compile ( if you had 
complied or from where did you took the release )
bq: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
This will come, when client did not find hadoop natives (hadoop.so,,) in class 
pathPlease check ,whethere you have hadoop natives in classpath or not..?
Normally, Hadoop will load natives from ${HADOOP_HOME}/lib/native or 
java.library.path..

Date: Sun, 19 Apr 2015 11:21:01 -0500
Subject: Re: Unable to load native-hadoop library
From: afsan...@gmail.com
To: user@hadoop.apache.org

are you using ubuntu? if yes look at JIRA HADOOP-10988
On Sat, Apr 18, 2015 at 2:00 PM, Mich Talebzadeh m...@peridale.co.uk wrote:
No I believe it was something to do with compilation. It is only a warning 
However, you should consider using hdfs dfs as opposed to hadoop fs I guess 
someone realised that “hadoop” is the name of eco system (HDFS + MapReduce)  
and hdfs is the actual file J which is more appropriate for a command syntax 
Mich Talebzadeh http://talebzadehmich.wordpress.com Author of the books A 
Practitioner’s Guide to Upgrading to Sybase ASE 15, ISBN 978-0-9563693-0-7. 
co-author Sybase Transact SQL Guidelines Best Practices, ISBN 
978-0-9759693-0-4Publications due shortly:Creating in-memory Data Grid for 
Trading Systems with Oracle TimesTen and Coherence CacheOracle and Sybase, 
Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly NOTE: 
The information in this email is proprietary and confidential. This message is 
for the designated recipient only, if you are not the intended recipient, you 
should destroy it immediately. Any information in this message shall not be 
understood as given or endorsed by Peridale Ltd, its subsidiaries or their 
employees, unless expressly so stated. It is the responsibility of the 
recipient to ensure that this email is virus free, therefore neither Peridale 
Ltd, its subsidiaries nor their employees accept any responsibility. From: 
Mahmood Naderan [mailto:nt_mahm...@yahoo.com] 
Sent: 18 April 2015 19:54
To: User
Subject: Unable to load native-hadoop library Hi,Regarding this warning WARN 
util.NativeCodeLoader: Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable It seems that the 
prebuild 32-bit binary is not compatible on the host's 64-bit architecture. 
Just want to know does it make sense? Is there any concern about the 
functionality? Regards,
Mahmood
  

RE: ipc.Client: Retrying connect to server

[Brahma Reddy Battula]

Hello Mahmood Naderan,
When client is trying to connect to server with configured port(and address) 
and server is not started with that port then client will retry ( and you will 
get following error..)..
I can able to trace that Namenode is not running From JPS report which you had 
posted..Please check the namenode logs ( location : 
/home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-namenode-tiger.out/log)
Date: Fri, 17 Apr 2015 08:22:12 +
From: nt_mahm...@yahoo.com
To: user@hadoop.apache.org
Subject: ipc.Client: Retrying connect to server

Hello,I have done all steps (as far as I know) to bring up the hadoop. However, 
I get the this error
15/04/17 12:45:31 INFO ipc.Client: Retrying connect to server: 
localhost/127.0.0.1:54310. Already tried 0 time(s).
There are a lot of threads and posts regarding this error and I tried them. 
However still stuck at this error :(
Can someone help me? What did I wrong?




Here are the configurations:
1) Hadoop configurations[mahmood@tiger hadoop-1.0.2]$ cat 
conf/mapred-site.xml?xml version=1.0??xml-stylesheet type=text/xsl 
href=configuration.xsl?!-- Put site-specific property overrides in this 
file. --configurationproperty  namemapred.job.tracker/name  
valuelocalhost:54311/value/propertyproperty  
namemapred.child.java.opts/name  
value-Xmx512m/value/property/configuration
[mahmood@tiger hadoop-1.0.2]$ cat conf/core-site.xml?xml 
version=1.0??xml-stylesheet type=text/xsl href=configuration.xsl?!-- 
Put site-specific property overrides in this file. --configurationproperty 
 namefs.default.name/name  
valuehdfs://localhost:54310/value/property/configuration
[mahmood@tiger hadoop-1.0.2]$ cat conf/hdfs-site.xml?xml 
version=1.0??xml-stylesheet type=text/xsl href=configuration.xsl?!-- 
Put site-specific property overrides in this file. --configurationproperty 
 namedfs.replication/name  value1/value/propertyproperty  
namehadoop.tmp.dir/name  
value/home/mahmood/bigdatabench/apache/hadoop-1.0.2/folders/tmp/value/propertyproperty
  namedfs.name.dir/name  
value/home/mahmood/bigdatabench/apache/hadoop-1.0.2/folders/name/value/propertyproperty
  namedfs.data.dir/name  
value/home/mahmood/bigdatabench/apache/hadoop-1.0.2/folders/data/value/property/configuration




2) Network configuration[root@tiger hadoop-1.0.2]# cat /etc/sysconfig/iptables# 
Firewall configuration written by system-config-firewall# Manual customization 
of this file is not recommended.*filter:INPUT ACCEPT [0:0]:FORWARD ACCEPT 
[0:0]:OUTPUT ACCEPT [0:0]-A INPUT -m state --state ESTABLISHED,RELATED -j 
ACCEPT-A INPUT -p icmp -j ACCEPT-A INPUT -i lo -j ACCEPT-A INPUT -m state 
--state NEW -m tcp -p tcp --dport 5901 -j ACCEPT-A INPUT -m state --state NEW 
-m tcp -p tcp --dport 80 -j ACCEPT-A INPUT -m state --state NEW -m tcp -p tcp 
--dport 22 -j ACCEPT-A INPUT -m state --state NEW -m tcp -p tcp --dport 2049 -j 
ACCEPT-A INPUT -m state --state NEW -m tcp -p tcp --dport 54310 -j ACCEPT-A 
INPUT -m state --state NEW -m tcp -p tcp --dport 54311 -j ACCEPT-A INPUT -j 
REJECT --reject-with icmp-host-prohibited-A FORWARD -j REJECT --reject-with 
icmp-host-prohibitedCOMMIT
[root@tiger hadoop-1.0.2]# /etc/init.d/iptables restartiptables: Flushing 
firewall rules: [  OK  ]iptables: Setting chains to 
policy ACCEPT: filter  [  OK  ]iptables: Unloading modules: 
  [  OK  ]iptables: Applying firewall rules:
 [  OK  ]
[mahmood@tiger hadoop-1.0.2]$ netstat -an | grep 54310[mahmood@tiger 
hadoop-1.0.2]$ netstat -an | grep 54311tcp0  0 
:::127.0.0.1:54311  :::*LISTENtcp  426  
0 :::127.0.0.1:54311  :::127.0.0.1:49639  ESTABLISHEDtcp
0  0 :::127.0.0.1:49639  :::127.0.0.1:54311  ESTABLISHED





3) Starting Hadoop[mahmood@tiger hadoop-1.0.2]$ stop-all.shWarning: 
$HADOOP_HOME is deprecated.stopping jobtrackerlocalhost: Warning: $HADOOP_HOME 
is deprecated.localhost:localhost: stopping tasktrackerno namenode to 
stoplocalhost: Warning: $HADOOP_HOME is deprecated.localhost:localhost: no 
datanode to stoplocalhost: Warning: $HADOOP_HOME is 
deprecated.localhost:localhost: stopping secondarynamenode
[mahmood@tiger hadoop-1.0.2]$ start-all.shWarning: $HADOOP_HOME is 
deprecated.starting namenode, logging to 
/home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-namenode-tiger.outlocalhost:
 Warning: $HADOOP_HOME is deprecated.localhost:localhost: starting datanode, 
logging to 
/home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-datanode-tiger.outlocalhost:
 Warning: $HADOOP_HOME is deprecated.localhost:localhost: starting 
secondarynamenode, logging to 
/home/mahmood/bigdatabench/apache/hadoop-1.0.2/libexec/../logs/hadoop-mahmood-secondarynamenode-tiger.outstarting
 jobtracker, logging to 

RE: ResourceLocalizationService: Localizer failed when running pi example

[Brahma Reddy Battula]

As Alexander Alten-Lorenz pointed, it mostly config 
issue(yarn.nodemanager.local-dirs or mapred.local.dir)..
can you able provide full logs..?
Bytheway NPE is handled in  Trunk ..Please check HADOOP-8436 for more details..

From: wget.n...@gmail.com
Subject: Re: ResourceLocalizationService: Localizer failed when running pi 
example
Date: Sun, 19 Apr 2015 17:59:13 +0200
To: user@hadoop.apache.org

As you said, that looks like a config issue. I would spot on the NM's local 
scratch dir (yarn.nodemanager.local-dirs).
But without a complete stack trace, its a blind call.
BR, AL
--mapredit.blogspot.com
On Apr 18, 2015, at 6:24 PM, Fernando O. fot...@gmail.com wrote:Hey All,
It's me again with another noob question: I deployed a cluster (HA mode) 
everything looked good but when I tried to run the pi example:
 bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 
16 100

the same error occurs if I try to generate data with teragen 1 
/test/data

2015-04-18 15:49:04,090 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Localizer failedjava.lang.NullPointerException   at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268)
   at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
  at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
  at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
  at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
  at 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:420)
 at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1075)

I'm guessing it's a configuration issue but I don't know what am I missing :S

  

RE: Unsubscribe

[Brahma Reddy Battula]

You can mailto user-unsubscr...@hadoop.apache.org

Date: Tue, 14 Apr 2015 09:12:07 -0700
Subject: Unsubscribe
From: preya.2.s...@gmail.com
To: user@hadoop.apache.org

unsubscribe


  

RE: hdfs ls command took more time to get response after update

[Brahma Reddy Battula]

Please send vendor specific questions to that vendor's support mechanism.

Since your issue appears to be with CDH, please use 
http://community.cloudera.com/http://community.cloudera.com/




Thanks  Regards

Brahma Reddy Battula





From: ZhuGe [t...@outlook.com]
Sent: Wednesday, April 08, 2015 8:41 AM
To: user@hadoop.apache.org
Subject: hdfs ls command took more time to get response after update

Hi all:
Recently, I updated my hadoop cluster from hadoop-2.0.0-cdh4.3.0 to  
hadoop-2.5.0-cdh5.2.0. It works fine, however, a small problem is, when i use 
the hadoop fs -ls command in the terminal to get the list of files in the hdfs, 
it took much more time(10+ sec) to get the response compared to 2-3 secs before 
i update the version of hadoop.( get is slow too)
Can any one exaplain a little bit of what might cause the problem or some 
configuration goes wrong?
Below is the log:

15/04/08 10:51:18 DEBUG util.Shell: setsid exited with exit code 0
15/04/08 10:51:18 DEBUG conf.Configuration: parsing URL 
jar:file:/data/dbcenter/cdh5/hadoop-2.5.0-cdh5.2.0/share/hadoop/common/hadoop-common-2.5.0-cdh5.2.0.jar!/core-default.xml
15/04/08 10:51:18 DEBUG conf.Configuration: parsing input stream 
sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@57316e85
15/04/08 10:51:18 DEBUG conf.Configuration: parsing URL 
file:/data/dbcenter/cdh5/hadoop-2.5.0-cdh5.2.0/etc/hadoop/core-site.xml
15/04/08 10:51:18 DEBUG conf.Configuration: parsing input stream 
java.io.BufferedInputStream@31818dbc
15/04/08 10:51:19 DEBUG lib.MutableMetricsFactory: field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, 
value=[Rate of successful kerberos logins and latency (milliseconds)], about=, 
type=DEFAULT, always=false, sampleName=Ops)
15/04/08 10:51:19 DEBUG lib.MutableMetricsFactory: field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, 
value=[Rate of failed kerberos logins and latency (milliseconds)], about=, 
type=DEFAULT, always=false, sampleName=Ops)
15/04/08 10:51:19 DEBUG lib.MutableMetricsFactory: field 
org.apache.hadoop.metrics2.lib.MutableRate 
org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with 
annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, 
value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops)
15/04/08 10:51:19 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group 
related metrics
15/04/08 10:51:19 DEBUG security.Groups:  Creating new Groups object
15/04/08 10:51:19 DEBUG util.NativeCodeLoader: Trying to load the custom-built 
native-hadoop library...
15/04/08 10:51:19 DEBUG util.NativeCodeLoader: Failed to load native-hadoop 
with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
15/04/08 10:51:19 DEBUG util.NativeCodeLoader: 
java.library.path=/data/dbcenter/cdh5/hadoop-2.5.0-cdh5.2.0/lib/native
15/04/08 10:51:19 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
15/04/08 10:51:19 DEBUG util.PerformanceAdvisory: Falling back to shell based
15/04/08 10:51:19 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group 
mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
15/04/08 10:51:19 DEBUG security.Groups: Group mapping 
impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; 
cacheTimeout=30; warningDeltaMs=5000
15/04/08 10:51:19 DEBUG security.UserGroupInformation: hadoop login
15/04/08 10:51:19 DEBUG security.UserGroupInformation: hadoop login commit
15/04/08 10:51:19 DEBUG security.UserGroupInformation: using local 
user:UnixPrincipal: test
15/04/08 10:51:19 DEBUG security.UserGroupInformation: UGI loginUser:test 
(auth:SIMPLE)
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: 
dfs.client.use.legacy.blockreader.local = false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = 
false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: 
dfs.client.domain.socket.data.traffic = false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
15/04/08 10:51:19 DEBUG hdfs.DFSClient: No KeyProvider found.
15/04/08 10:51:19 DEBUG hdfs.HAUtil: No HA service delegation token found for 
logical URI hdfs://tccluster:8020
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: 
dfs.client.use.legacy.blockreader.local = false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = 
false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: 
dfs.client.domain.socket.data.traffic = false
15/04/08 10:51:19 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
15/04/08 10:51:19 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
15/04/08 10:51:19 DEBUG ipc.Server

unsubscribe

[Brahma Reddy Battula]

Kindly Send email to user-unsubscr...@hadoop.apache.org

Subject: Re: Hadoop 2.6 issue
To: user@hadoop.apache.org
From: rapen...@in.ibm.com
Date: Thu, 2 Apr 2015 09:02:09 +0530


Please un subscribe me from this list.



Regards,

Ravi Prasad Pentakota

India Software Lab, IBM Software Group

Phone: +9180-43328520  Mobile: 919620959477 

e-mail:rapen...@in.ibm.com






Kumar Jayapal ---04/02/2015 07:50:05 AM---$which java make sure the paths are 
valid for your installation (change if using 32bit



From:   Kumar Jayapal kjayapa...@gmail.com

To: user@hadoop.apache.org

Cc: Anand Murali anand_vi...@yahoo.com

Date:   04/02/2015 07:50 AM

Subject:Re: Hadoop 2.6 issue







$which java



make sure the paths are valid for your installation (change if using 32bit 
version): 

/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java



/usr/lib/jvm/java-6-openjdk-amd64/bin/javac
Setup update-alternatives:

sudo update-alternatives --install /usr/bin/java java 
/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1

sudo update-alternatives --install /usr/bin/javac javac 
/usr/lib/jvm/java-6-openjdk-amd64/bin/javac 1



sudo update-alternatives --set java 
/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java

sudo update-alternatives --set javac /usr/lib/jvm/java-6-openjdk-amd64/bin/javac

Alternatively, make sure the correct version is checked for both Java and 
compiler:

sudo update-alternatives --config java

sudo update-alternatives --config javac

List the installed Java alternatives with:

sudo update-alternatives --list java

sudo update-alternatives --list javac



On Wed, Apr 1, 2015 at 10:35 AM, Ravindra Kumar Naik ravin.i...@gmail.com 
wrote:

Hi,



Creating batch program will not have the same effect. If you put the variables 
in /etc/environment then it will be available to all users on the operating 
system. HDFS doesn't run with root privileges.

You need to open the application with sudo or with root privileges to modify it.

e.g. If you are using vi editor then its just sudo vim /etc/environment 
(similar, if you are using other editors) and add environment variables there.





On Wed, Apr 1, 2015 at 7:38 PM, Anand Murali anand_vi...@yahoo.com wrote:
Mr. Ravindra:



This is visible, however I am unable to modify it, eventhough I have admin 
priveleges. I am new to the Linux environment. Shall be glad if you did advise. 
However, as I told you earlier, I have created a batch program which contains, 
JAVA_HOME setting, HADOOP_INSTALL setting and PATH setting. I have rfun this 
file but I am still unable to start the daemons. I am following Tom Whyte's 
-Hadoop definitive Guide book instructions on how to install Hadoop.



at $hadoop version works. I am able to format namenode, but fail to start 
daemons.



Reply most welcome.



Thanks

 

Anand Murali  

11/7, 'Anand Vihar', Kandasamy St, Mylapore

Chennai - 600 004, India

Ph: (044)- 28474593/ 43526162 (voicemail)







On Wednesday, April 1, 2015 7:04 PM, Ravindra Kumar Naik ravin.i...@gmail.com 
wrote:





Are you sure that its not there, could you please check the output of this 
command



ls /etc/env*







On Wed, Apr 1, 2015 at 6:55 PM, Anand Murali anand_vi...@yahoo.com wrote:
Mr. Ravindra:



I am using Ubuntu 14. Can you please provide the full path. I am logged in as 
root and it is not found in /etc. In any case what you have suggested I have 
tried creating a batch file and it does not work in my installation.



Thanks





 

Anand Murali  

11/7, 'Anand Vihar', Kandasamy St, Mylapore

Chennai - 600 004, India

Ph: (044)- 28474593/ 43526162 (voicemail)







On Wednesday, April 1, 2015 6:50 PM, Ravindra Kumar Naik ravin.i...@gmail.com 
wrote:





I meant /etc/environment. It should be present if you are using Ubuntu.



Regards,

Ravindra



On Wed, Apr 1, 2015 at 6:39 PM, Anand Murali anand_vi...@yahoo.com wrote:
Mr. Ravindra 



I dont find any etc/environment. Can you be more specific please. I have done 
whatever you are saying in a user created batch program and run it, followed by 
running hadoop-env.sh and it still does not work.



Thanks

 

Anand Murali  

11/7, 'Anand Vihar', Kandasamy St, Mylapore

Chennai - 600 004, India

Ph: (044)- 28474593/ 43526162 (voicemail)







On Wednesday, April 1, 2015 6:10 PM, Ravindra Kumar Naik ravin.i...@gmail.com 
wrote:





Hi,



If you are using Ubuntu then add these lines to /etc/environment 

JAVA_HOME=actual path to jdk

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:$JAVA_HOME/bin



Please put the actual path to JDK in the first line.



Regards,

Ravindra





On Wed, Apr 1, 2015 at 5:50 PM, roland.depratti roland.depra...@cox.net wrote:
Anand,



Sorry about that, I was assuming Redhat/Centos.



For Ubuntu, try sudo update-alternatives --config java.







Sent from my Verizon Wireless 4G LTE smartphone





 Original message 

From: Anand Murali anand_vi...@yahoo.com 

Date: 04/01/2015 7:22 AM (GMT-05:00) 

To: 

RE: How to know when datanode are marked dead by namenode

[Brahma Reddy Battula]

HI

The NameNode updates this detail after 10.30 minutes by default. You can see 
the dead and live datanodes at that time.

It computes this heartbeatExpireInterval time by the following formula

  heartbeatExpireInterval = 2 * heartbeatRecheckInterval +
  10 * 1000* heartbeatInterval

where heartbeatRecheckInterval is defined by the configuration 
dfs.namenode.heartbeat.recheck-interval which is 5 minutes by default and 
heartbeatInterval by dfs.heartbeat.interval which is 3 seconds by default.

Hence
heartbeatExpireInterval = 10.30 minutes


SO If you want keep more time, you can configure 
dfs.namenode.heartbeat.recheck-interval based one your requirement..



Thanks  Regards

Brahma Reddy Battula





From: Himawan Mahardianto [mahardia...@ugm.ac.id]
Sent: Saturday, March 28, 2015 4:42 PM
To: user@hadoop.apache.org
Subject: How to know when datanode are marked dead by namenode

Hi guys I'm newbie here, do you know how to time calculation when datanode are 
marked dead by namenode, what parameters on HDFS-SITE.xml should I look for to 
calculate it, and how can I reduce dead time from default 10 minutes to 5 
minutes or increase it to 20 minutes?
Thank's before

best regards
Himawan Mahardianto

RE: Hadoop 2.6.0 Error

[Brahma Reddy Battula]

Instead of exporting the JAVA_HOME, Please set JAVA_HOME in system level ( like 
putting in /etc/profile...)

For more details please check the following jira.

https://issues.apache.org/jira/browse/HADOOP-11538




Thanks  Regards

 Brahma Reddy Battula


From: Anand Murali [anand_vi...@yahoo.com]
Sent: Wednesday, March 25, 2015 11:23 AM
To: User Hadoop
Subject: Hadoop 2.6.0 Error

Dear All:

Request help/advise as I am unable to start Hadoop. Performed follow steps in 
Ubuntu 14.10

1. ssh localhost
2. Did following exports in user defined hadoop.sh and ran it succesfully
1. EXPORT JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
2. EXPORT HADOOP_INSTALL=/home/anand_vihar/hadoop-2.6.0
3. EXPORT PATH=:$PATH:$HADOOP_INSTALL/sbin:$HADOOP_INSTALL/bin
3. Tested hadoop version succesfully
4. Ran $hadoop namenode -format successfully
5. Modified core-site.xml, hdfs-site.xml and mapred-site.xml to 
pseudo-distributed mode in /home/anand_vihar/conf directory
6. Ran $start-dfs.sh --config /home/anand_vihar/conf

Got error JAVA_HOME not set and slaves not found in /conf. If I echo $JAVA_HOME 
it is pointing to /usr/lib/jvm/java-7-openjdk-amd6, correctly as set. Help 
appreciated.

Thanks

Regards,

Anand Murali
11/7, 'Anand Vihar', Kandasamy St, Mylapore
Chennai - 600 004, India
Ph: (044)- 28474593/ 43526162 (voicemail)

RE: Hadoop 2.6.0 Error

[Brahma Reddy Battula]

HI

Ideally it should take effect , if you configure in .profile or hadoop-env.sh..

As you told that you set in .profile ( hope you did source ~/.profile ),,,

 did you verify that take effect..?  ( by checking echo $JAVA_HOME,, or 
jps..etc )...




Thanks  Regards

Brahma Reddy Battula





From: Anand Murali [anand_vi...@yahoo.com]
Sent: Wednesday, March 25, 2015 1:30 PM
To: user@hadoop.apache.org; Anand Murali
Subject: Re: Hadoop 2.6.0 Error

Dear All:

Even after setting JAVA_HOME in .profile I get

JAVA_HOME is not set and could not be found -error.


If anyone of you know of a more stable version please do let me know.

Thanks,

Anand Murali
11/7, 'Anand Vihar', Kandasamy St, Mylapore
Chennai - 600 004, India
Ph: (044)- 28474593/ 43526162 (voicemail)



On Wednesday, March 25, 2015 12:57 PM, Anand Murali anand_vi...@yahoo.com 
wrote:


Dear Mr. Bhrama Reddy:

Should I type

SET JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64


in root (profile) or at user level (.profile). Reply most welcome

Thanks

Regards


Anand Murali
11/7, 'Anand Vihar', Kandasamy St, Mylapore
Chennai - 600 004, India
Ph: (044)- 28474593/ 43526162 (voicemail)



On Wednesday, March 25, 2015 12:37 PM, Anand Murali anand_vi...@yahoo.com 
wrote:


Dear All:

I get this error shall try setting JAVA_HOME in .profile

Starting namenodes on [localhost]
localhost: Error: JAVA_HOME is not set and could not be found.
cat: /home/anand_vihar/hadoop-2.6.0/conf/slaves: No such file or directory
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Error: JAVA_HOME is not set and could not be found.
anand_vihar@Latitude-E5540:~/hadoop-2.6.0/sbin$

Thanks

Anand Murali
11/7, 'Anand Vihar', Kandasamy St, Mylapore
Chennai - 600 004, India
Ph: (044)- 28474593/ 43526162 (voicemail)



On Wednesday, March 25, 2015 12:22 PM, Brahma Reddy Battula 
brahmareddy.batt...@huawei.com wrote:


Instead of exporting the JAVA_HOME, Please set JAVA_HOME in system level ( like 
putting in /etc/profile...)

For more details please check the following jira.

https://issues.apache.org/jira/browse/HADOOP-11538



Thanks  Regards
 Brahma Reddy Battula


From: Anand Murali [anand_vi...@yahoo.com]
Sent: Wednesday, March 25, 2015 11:23 AM
To: User Hadoop
Subject: Hadoop 2.6.0 Error

Dear All:

Request help/advise as I am unable to start Hadoop. Performed follow steps in 
Ubuntu 14.10

1. ssh localhost
2. Did following exports in user defined hadoop.sh and ran it succesfully
1. EXPORT JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
2. EXPORT HADOOP_INSTALL=/home/anand_vihar/hadoop-2.6.0
3. EXPORT PATH=:$PATH:$HADOOP_INSTALL/sbin:$HADOOP_INSTALL/bin
3. Tested hadoop version succesfully
4. Ran $hadoop namenode -format successfully
5. Modified core-site.xml, hdfs-site.xml and mapred-site.xml to 
pseudo-distributed mode in /home/anand_vihar/conf directory
6. Ran $start-dfs.sh --config /home/anand_vihar/conf

Got error JAVA_HOME not set and slaves not found in /conf. If I echo $JAVA_HOME 
it is pointing to /usr/lib/jvm/java-7-openjdk-amd6, correctly as set. Help 
appreciated.

Thanks

Regards,

Anand Murali
11/7, 'Anand Vihar', Kandasamy St, Mylapore
Chennai - 600 004, India
Ph: (044)- 28474593/ 43526162 (voicemail)

RE: journal node shared edits directory should be present on HDFS or NAS or anything else?

[Brahma Reddy Battula]

Hello Chandrashekhar,

Active namenode will write to require shared storage and will not write to 
HDFS.. Please check following docs for reference



When Sharedstorage is Journalnode:


property
  namedfs.namenode.shared.edits.dir/name
  
valueqjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/mycluster/value
/property

http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html



When Sharedstorage is NFS:


property
  namedfs.namenode.shared.edits.dir/name
  valuefile:///mnt/filer1/dfs/ha-name-dir-shared/value
/property


http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html






Thanks  Regards

 Brahma Reddy Battula




From: Chandrashekhar Kotekar [shekhar.kote...@gmail.com]
Sent: Thursday, February 12, 2015 5:01 PM
To: user@hadoop.apache.org
Subject: journal node shared edits directory should be present on HDFS or NAS 
or anything else?

Hi,

I am  trying to configure name node HA and I want to further configure 
automatic fail over. I am confused about 'dfs.namenode.shared.edits.dir' 
configuration.

Documentation says that active namde node writes to shared storage. I would 
like to know if this means that name nodes write it on HDFS or do they require 
shared storage like NAS or SAN or something else.


Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455

RE: hadoop cluster with non-uniform disk spec

[Brahma Reddy Battula]

Hello daemeon reiydelle


Is the policy set to 
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?

Yes, you need to set this policy which will balance among the disks

@Chen Song

following settings controls what percentage of new block allocations will be 
sent to volumes with more available disk space than others

dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 
21474836480 (20G)
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction
 = 0.85f


Did you set while startup the cluster..?



Thanks  Regards

 Brahma Reddy Battula





From: daemeon reiydelle [daeme...@gmail.com]
Sent: Thursday, February 12, 2015 12:02 PM
To: user@hadoop.apache.org
Cc: Ravi Prakash
Subject: Re: hadoop cluster with non-uniform disk spec

What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assuming 
you are on a current version of Hadoop)? Is the policy set to 
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy?



...
“Life should not be a journey to the grave with the intention of arriving 
safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of 
smoke,
thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!”
- Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Wed, Feb 11, 2015 at 2:23 PM, Chen Song 
chen.song...@gmail.commailto:chen.song...@gmail.com wrote:
Hey Ravi

Here are my settings:
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 
21474836480 (20G)
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction
 = 0.85f

Chen


On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash 
ravi...@ymail.commailto:ravi...@ymail.com wrote:
Hi Chen!

Are you running the balancer? What are you setting 
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction
to?




On Wednesday, February 11, 2015 7:44 AM, Chen Song 
chen.song...@gmail.commailto:chen.song...@gmail.com wrote:


We have a hadoop cluster consisting of 500 nodes. But the nodes are not uniform 
in term of disk spaces. Half of the racks are newer with 11 volumes of 1.1T on 
each node, while the other half have 5 volume of 900GB on each node.

dfs.datanode.fsdataset.volume.choosing.policy is set to 
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

It winds up with the state of half of nodes are full while the other half 
underutilized. I am wondering if there is a known solution for this problem.

Thank you for any suggestions.

--
Chen Song






--
Chen Song

RE: journal node shared edits directory should be present on HDFS or NAS or anything else?

[Brahma Reddy Battula]

Hello Chandrashekhar,

Yes, you need to configure the shared storage(Active namenode writes to shared 
storage and Standby NN will read).

 Please check following mail for configuration..Shared storage can be 
Journalnode(which is one process and come along with hadoop package, check 
following reference for more details)/NFS/BOOKKeeper





Thanks  Regards

 Brahma Reddy Battula





From: Chandrashekhar Kotekar [shekhar.kote...@gmail.com]
Sent: Thursday, February 12, 2015 5:31 PM
To: user@hadoop.apache.org
Subject: Re: journal node shared edits directory should be present on HDFS or 
NAS or anything else?

Hi Brahma Reddy,

Thanks for the quick answer. It explains a lot but I have one more question. 
Maybe it is a stupid question but, required shared storage means active name 
node will write to its local disk? Do I need to configure or use any shared 
storage like NAS or SAN array or S3 storage for this purpose?


Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455

On Thu, Feb 12, 2015 at 5:08 PM, Brahma Reddy Battula 
brahmareddy.batt...@huawei.commailto:brahmareddy.batt...@huawei.com wrote:
Hello Chandrashekhar,

Active namenode will write to require shared storage and will not write to 
HDFS.. Please check following docs for reference



When Sharedstorage is Journalnode:


property
  namedfs.namenode.shared.edits.dir/name
  
valueqjournal://node1.example.com:8485http://node1.example.com:8485;node2.example.com:8485http://node2.example.com:8485;node3.example.com:8485/myclusterhttp://node3.example.com:8485/mycluster/value
/property

http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html



When Sharedstorage is NFS:


property
  namedfs.namenode.shared.edits.dir/name
  valuefile:///mnt/filer1/dfs/ha-name-dir-shared/value
/property


http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html






Thanks  Regards

 Brahma Reddy Battula




From: Chandrashekhar Kotekar 
[shekhar.kote...@gmail.commailto:shekhar.kote...@gmail.com]
Sent: Thursday, February 12, 2015 5:01 PM
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
Subject: journal node shared edits directory should be present on HDFS or NAS 
or anything else?

Hi,

I am  trying to configure name node HA and I want to further configure 
automatic fail over. I am confused about 'dfs.namenode.shared.edits.dir' 
configuration.

Documentation says that active namde node writes to shared storage. I would 
like to know if this means that name nodes write it on HDFS or do they require 
shared storage like NAS or SAN or something else.


Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455tel:%2B91%208600011455

RE: Failed to start datanode due to bind exception

[Brahma Reddy Battula]

Hello Rajesh


I think, you might have configured dfs.domain.socket.path as 
/var/run/hdfs-sockets/datanode

Actually ,This is a path to a UNIX domain socket that will be used for 
communication between the DataNode and local HDFS clients. If the string 
_PORT is present in this path, it will be replaced by the TCP port of the 
DataNode.

Ideally if some port present only , you will get that error.please 
re-checkonce..

if you delete /var/run/hdfs-sockets/datanode (worst condition, if it is 
corrupted) and start the datanode.




Thanks  Regards

 Brahma Reddy Battula





From: Rajesh Thallam [rajesh.thal...@gmail.com]
Sent: Wednesday, February 11, 2015 12:09 AM
To: user@hadoop.apache.org
Subject: Re: Failed to start datanode due to bind exception

There are no contents in the hdfs-sockets directory
Apache Hadoop Base version if 2.5.0 (using CDH 5.3.0)

On Tue, Feb 10, 2015 at 10:24 AM, Ted Yu 
yuzhih...@gmail.commailto:yuzhih...@gmail.com wrote:
The exception came from DomainSocket so using netstat wouldn't reveal the 
conflict.

What's the output from:
ls -l /var/run/hdfs-sockets/datanode

Which hadoop release are you using ?

Cheers

On Tue, Feb 10, 2015 at 10:12 AM, Rajesh Thallam 
rajesh.thal...@gmail.commailto:rajesh.thal...@gmail.com wrote:

I have been repeatedly trying to start datanode but it fails with bind 
exception saying address is already in use even though port is free

I used below commands to check

netstat -a -t --numeric-ports -p | grep 500



I have overridden default port 50070 with 50081 but the issue still persists.

Starting DataNode with maxLockedMemory = 0
Opened streaming server at /172.19.7.160:50081http://172.19.7.160:50081
Balancing bandwith is 10485760 bytes/s
Number threads for balancing is 5
Waiting for threadgroup to exit, active threads is 0
Shutdown complete.
Exception in secureMain
java.net.BindException: bind(2) error: Address already in use when trying to 
bind to '/var/run/hdfs-sockets/datanode'
at org.apache.hadoop.net.unix.DomainSocket.bind0(Native Method)
at 
org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:191)
at 
org.apache.hadoop.hdfs.net.DomainPeerServer.init(DomainPeerServer.java:40)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:907)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:873)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1066)
at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:411)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2297)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2184)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2231)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2407)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2431)
Exiting with status 1

hdfs-site.xml

  property
namedfs.datanode.address/name
valuehostname.dc.xx.org:50010http://hostname.dc.xx.org:50010/value
  /property
  property
namedfs.datanode.ipc.address/name
valuehostname.dc.xx.org:50020http://hostname.dc.xx.org:50020/value
  /property
  property
namedfs.datanode.http.address/name
valuehostname.dc.xx.org:50075http://hostname.dc.xx.org:50075/value
  /property

Regards,
RT




--
Cheers,
RT

RE: Hadoop 2.6.0: FileSystem file:/// is not a distributed file system

[Brahma Reddy Battula]

Hi Dong,

HADOOP_CONF_DIR might be referring to default..you can export HADOOP_CONF_DIR 
where following configuration files are present..


Thanks  Regards

Brahma Reddy Battula



From: Dan Dong [dongda...@gmail.com]
Sent: Saturday, December 13, 2014 3:43 AM
To: user@hadoop.apache.org
Subject: Hadoop 2.6.0: FileSystem file:/// is not a distributed file system

Hi,
  I installed Hadoop2.6.0 on my cluster with 2 nodes, I got the following error 
when I run:
$hadoop dfsadmin -report
FileSystem file:/// is not a distributed file system

What this mean? I have set it in core-site.xml already:
property
  namefs.defaultFS/name
  valuehdfs://master-node:9000/value
/property

and in hdfs-site.xml:
property
  namedfs.namenode.name.dir/name
  valuefile:/home/dong/hadoop-2.6.0-dist/dfs/name/value
  finaltrue/final
/property
property
  namedfs.dataname.data.dir/name
  valuefile:/home/dong/hadoop-2.6.0-dist/dfs/data/value
  finaltrue/final
/property

The java process are running on master as:
10479 SecondaryNameNode
10281 NameNode
10628 ResourceManager

and on slave:
22870 DataNode
22991 NodeManager

Any hints? Thanks!

Cheers,
Dan

RE: how to increate data node expire time

[Brahma Reddy Battula]

Hi Henry


The NameNode updates this detail after 10.30 minutes by default. You can see 
the dead and live datanodes at that time.

It computes this heartbeatExpireInterval time by the following formula

  heartbeatExpireInterval = 2 * heartbeatRecheckInterval +
  10 * 1000* heartbeatInterval

where heartbeatRecheckInterval is defined by the configuration 
dfs.namenode.heartbeat.recheck-interval which is 5 minutes by default and 
heartbeatInterval by dfs.heartbeat.interval which is 3 seconds by default.

Hence
heartbeatExpireInterval = 10.30 minutes


SO If you want keep more time, you can configure 
dfs.namenode.heartbeat.recheck-interval based one your requirement..


Thanks  Regards

Brahma Reddy Battula



From: Henry Hung [ythu...@winbond.com]
Sent: Tuesday, November 25, 2014 8:05 AM
To: user@hadoop.apache.org
Subject: how to increate data node expire time

Hi All,

In Hadoop 2.2.0, how to increase the data node expire time before name node 
list it as a dead node?

I want to shut down a data node for 30 minutes, and I don’t want the name node 
to list it as dead node to prevent the re-replication process.

Best regards,
Henry


The privileged confidential information contained in this email is intended for 
use only by the addressees as indicated by the original sender of this email. 
If you are not the addressee indicated in this email or are not responsible for 
delivery of the email to such a person, please kindly reply to the sender 
indicating this fact and delete all copies of it from your computer and network 
server immediately. Your cooperation is highly appreciated. It is advised that 
any unauthorized use of confidential information of Winbond is strictly 
prohibited; and any information in this email irrelevant to the official 
business of Winbond shall be deemed as neither given nor endorsed by Winbond.

RE: Configure Rack Numbers

[Brahma Reddy Battula]

Hi Navaz,

you have to configure the following two properties in namenode(after that you 
need to restart the namenode).


property
  nametopology.node.switch.mapping.impl/name
  valueorg.apache.hadoop.net.ScriptBasedMapping/value
  description The default implementation of the DNSToSwitchMapping. It
invokes a script specified in topology.script.file.name to resolve
node names. If the value for topology.script.file.name is not set, the
default value of DEFAULT_RACK is returned for all node names.
  /description
/property

property
  nametopology.script.file.name/name
  value/path/to/topo.sh/value
  description The script name that should be invoked to resolve DNS names to
NetworkTopology names. Example: the script would take host.foo.bar as an
argument, and return /rack1 as the output.
  /description
/property


Example script file.


topo.sh
===

#!/bin/bash

python TOPOLOGY_SCRIPT_HOME/topology.py $@


topology.py
===

import sys
from string import join

DEFAULT_RACK = '/default/rack0';

RACK_MAP = { '208.94.2.10' : '/datacenter1/rack0',
 '1.2.3.4' : '/datacenter1/rack1',
 '1.2.3.5' : '/datacenter1/rack1',
 '1.2.3.6' : '/datacenter1/rack1',

 '10.2.3.4' : '/datacenter1/rack2',
 '10.2.3.4' : '/datacenter1/rack2'
}

if len(sys.argv)==1:
print DEFAULT_RACK
else:
print join([RACK_MAP.get(i, DEFAULT_RACK) for i in sys.argv[1:]], )


Please check the following link for more details.


https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf




Thanks  Regards

 Brahma Reddy Battula



HUAWEI TECHNOLOGIES INDIA PVT.LTD.
Ground,12 floors,Solitaire,
139/26,Amarjyoti Layout,Intermediate Ring Road,Domlur
Bangalore - 560 071 , India
Tel : +91- 80- 3980 9600  Ext No: 4905
 Fax : +91-80-41118578


From: Abdul Navaz [navaz@gmail.com]
Sent: Monday, November 17, 2014 4:48 AM
To: user@hadoop.apache.org
Subject: Configure Rack Numbers


Hello,

I have hadoop cluster with 9 nodes. All belongs to /default racks. But I want 
the setup something similar to this.

(All are in same subnets)

 Rack 0: DataNode1,Datanode2,DataNode3 and top of rack switch1.
 Rack 1: DataNode4,Datanode5,DataNode6 and top of rack switch2.
 Rack 3: DataNode7,Datanode8,DataNode9 and top of rack switch3.


I am trying to check the Hadoop rack awareness and how it copies the single 
block of data in one rack and replicas in some other rack. I want to analyse 
some network performance from this.

So how can we separate this DNs based on rack numbers. Where can we configure 
this rack numbers and say this DN belongs to this rack number.


Thanks  Regards,

Abdul Navaz

RE: issue about pig can not know HDFS HA configuration

[Brahma Reddy Battula]

Hello Jagannath,

Below exception will come when pigclient not able find the hdfs 
configuraions..you need to do following..


Set the PIG_CLASSPATH environment variable to the location of the cluster 
configuration directory (the directory that contains the core-site.xml, 
hdfs-site.xml and mapred-site.xml files):

  1.

export PIG_CLASSPATH=/mycluster/conf


  2.  Set the HADOOP_CONF_DIR environment variable to the location of the 
cluster configuration directory:

export HADOOP_CONF_DIR=/mycluster/conf





Thanks  Regards

Brahma Reddy Battula





From: Jagannath Naidu [jagannath.na...@fosteringlinux.com]
Sent: Wednesday, November 05, 2014 5:11 PM
To: user@hadoop.apache.org
Subject: Re: issue about pig can not know HDFS HA configuration



On 5 November 2014 14:49, ch huang 
justlo...@gmail.commailto:justlo...@gmail.com wrote:
hi,maillist:
   i set namenode HA in my HDFS cluster,but seems pig can not know it ,why?

2014-11-05 14:34:54,710 [JobControl] INFO  
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area 
file:/tmp/hadoop-root/mapred/staging/root1861403840/.staging/job_local1861403840_0001
2014-11-05 14:34:54,716 [JobControl] WARN  
org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException 
as:root (auth:SIMPLE) 
cause:org.apache.pig.backend.executionengine.ExecException: ERROR 2118: 
java.net.UnknownHostException: develop

unknown host exception, this can be the issue. Check that the host is 
discoverable either form dns or from hosts.

2014-11-05 14:34:54,717 [JobControl] INFO  
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob - 
PigLatin:DefaultJobName got an error while submitting
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: 
java.net.UnknownHostException: develop
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:288)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at 
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
at 
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:744)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: 
develop
at 
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:237)
at 
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:141)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:576)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:521)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at 
org.apache.hcatalog.mapreduce.HCatBaseInputFormat.setInputPath(HCatBaseInputFormat.java:326)
at 
org.apache.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:127)
at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
... 18 more
Caused by: java.net.UnknownHostException: develop
... 33 more




--

Jaggu Naidu

RE: HDFS multiple dfs_data_dir disbalance

[Brahma Reddy Battula]

does /data1 is having non-hadoop data..? Please check for same..
check admin report(hdfs dfsadmin -report) and fsck report (hdfs fsck /)

Thinking that following might not 
a) /data/2 is not having the permission to write data where volumes tolerated 
configured as 1
b) /data/2 is not added after some time



Thanks  Regards
Brahma Reddy Battula

From: Georgi Ivanov [iva...@vesseltracker.com]
Sent: Wednesday, October 22, 2014 5:17 PM
To: user@hadoop.apache.org
Subject: HDFS multiple dfs_data_dir disbalance

Hi,
My cluster is configured with 2 data dirs.
/data/1
/data/2

Usually hadoop is balancing the utilization of these dirs.
Now i have one node where /data/1 is 100% full and /data/2 is not.

Is there anything i can do about this, as this results in failed
mapppers/rdecers ?

Georgi

RE: Error executing job in YARN. Exception Shell$ExitCodeException

[Brahma Reddy Battula]

Hi gortiz

Please have a look at application master logs(you can get RM UI or JHS (if 
aggeration enabled) or yarn-nodemanger-logs-dir (if aggeration is disabled) ) 
and Nodemanager logs to get the exact cause..



Thanks  Regards
Brahma Reddy Battula





From: gortiz [gor...@pragsis.com]
Sent: Wednesday, October 15, 2014 5:25 PM
To: user@hadoop.apache.org
Subject: Error executing job in YARN. Exception Shell$ExitCodeException

I'm trying to execute a work in YARN but I get an error. I have checked
the yarn-site.xml, the classpath seems to be alright.
I read https://issues.apache.org/jira/browse/YARN-1473 as well, but it
didn't work to me either.

Container for appattempt_1413373500815_0001_02 exited with exitCode:
1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
 at org.apache.hadoop.util.Shell.run(Shell.java:424)
 at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
 at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
 at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)


What else could it be happened?
AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo 
es privada y confidencial y va dirigida exclusivamente a su destinatario. 
Pragsis informa a quien pueda haber recibido este correo por error que contiene 
información confidencial cuyo uso, copia, reproducción o distribución está 
expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este 
correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su 
eliminación sin copiarlo, imprimirlo o utilizarlo de ningún 
modo.\nCONFIDENTIALITY WARNING.\nThis message and the information contained in 
or attached to it are private and confidential and intended exclusively for the 
addressee. Pragsis informs to whom it may receive it in error that it contains 
privileged information and its use, copy, reproduction or distribution is 
prohibited. If you are not an intended recipient of this E-mail, please notify 
the sender, delete it and do not read, act upon, print, disclose, c!
opy, reta
 in or redistribute any portion of this E-mail.

RE: Not able to update jar

[Brahma Reddy Battula]

Hi Patcharee

Did you put the updated jar under $HADOOP_HOME/share/hadoop/common/lib


Please send the exception which you are getting while executing..


and syntax is $HADOOP_HOME/bin/hadoop jar jar name class name arg0 arg1





Thanks  Regards



Brahma Reddy Battula





From: Patcharee Thongtra [patcharee.thong...@uni.no]
Sent: Friday, October 10, 2014 6:02 PM
To: user@hadoop.apache.org
Subject: Re: Not able to update jar

I am using hadoop 2.2.0

Patcharee

On 10/10/2014 02:21 PM, Ted Yu wrote:
Which Hadoop release are you using ?

Cheers

On Oct 10, 2014, at 5:09 AM, Patcharee Thongtra 
patcharee.thong...@uni.nomailto:patcharee.thong...@uni.no wrote:

Yes I meant hadoop jar myjar.jar package.classname arg0 arg1, but the problem 
is the latest version of myjar.jar has not been executed.

Patcharee

On 10/10/2014 01:50 PM, vivek wrote:

I think the syntax is hadoop jar myjar.jar package.classname arg0 arg1

On Fri, Oct 10, 2014 at 7:42 AM, Patcharee Thongtra 
patcharee.thong...@uni.nomailto:patcharee.thong...@uni.no wrote:
Hi,

I am not able to update my jar, it seems it has been cached somewhere

I run hadoop -jar myjar.jar arg0 arg1

How can I fix this?

Patcharee



--







Thanks and Regards,

VIVEK KOUL

RE: How to find extent of data node imbalance ?

[Brahma Reddy Battula]

HI Manoj,

You can check from namenode web UI also apart from what you mentioned..



http://NameNode-host:httpPort/dfsnodelist.jsp?whatNodes=LIVEhttp://10.18.51.157:25000/dfsnodelist.jsp?whatNodes=LIVE





Thanks  Regards



Brahma Reddy Battula





From: Manoj Samel [manojsamelt...@gmail.com]
Sent: Saturday, October 11, 2014 2:26 AM
To: user@hadoop.apache.org
Subject: Re: How to find extent of data node imbalance ?

Reposting ...

One option is to do hdfs dfsadmin -report and see DFS Used% on each data 
node and then compute the extent of imbalance across nodes. Is there any other 
way ?

Thanks,

On Wed, Oct 8, 2014 at 3:33 PM, Manoj Samel 
manojsamelt...@gmail.commailto:manojsamelt...@gmail.com wrote:
Hi,

Before running hadoop rebalancer, it is possible to find the extent to which 
the data nodes are unbalanced ?

Thanks,

RE: DFS Used V/S Non DFS Used

[Brahma Reddy Battula]

Hi Manoj


Non DFS used is any data in the filesystem of the data node(s) that isn't in 
dfs.datanode.data.dirs.

This would include log files, mapreduce shuffle output and local copies of data 
files (if you put them on a data node).

Use du or a similar tool to see whats taking up the space in your filesystem..


Non DFS used is calculated by following formula:

Non DFS Used = Configured Capacity - Remaining Space - DFS Used

It is still confusing, at least for me.

Because Configured Capacity = Total Disk Space - Reserved Space.

So Non DFS used = ( Total Disk Space - Reserved Space) - Remaining Space - DFS 
Used

Let's take a example. Assuming I have 100 GB disk, and I set the reserved space 
(dfs.datanode.du.reserved) to 30 GB.

In the disk, the system and other files used up to 40 GB, DFS Used 10 GB. If 
you run df -h , you will see the available space is 50GB for that disk volume.

In HDFS web UI, it will show

Non DFS used = 100GB(Total) - 30 GB( Reserved) - 10 GB (DFS used) - 
50GB(Remaining) = 10 GB

So it actually means, you initially configured to reserve 30G for non dfs 
usage, and 70 G for HDFS. However, it turns out non dfs usage exceeds the 30G 
reservation and eat up 10 GB space which should belongs to HDFS!

The term Non DFS used should really be renamed to something like How much 
configured DFS capacity are occupied by non dfs use

And one should stop try to figure out why the non dfs use are so high inside 
hadoop.

One useful command is lsof | grep delete, which will help you identify those 
open file which has been deleted. Sometimes, Hadoop processes (like hive, yarn, 
and mapred and hdfs) may hold reference to those already deleted files. And 
these references will occupy disk space.

Also du -hsx * | sort -rh | head -10 helps list the top ten largest folders.





Thanks  Regards



Brahma Reddy Battula



HUAWEI TECHNOLOGIES INDIA PVT.LTD.
Ground,12 floors,Solitaire,
139/26,Amarjyoti Layout,Intermediate Ring Road,Domlur
Bangalore - 560 071 , India
Tel : +91- 80- 3980 9600  Ext No: 4905
Mobile : +91   9620022006
Fax : +91-80-41118578


From: Manoj Samel [manojsamelt...@gmail.com]
Sent: Saturday, October 11, 2014 3:08 AM
To: user@hadoop.apache.org
Subject: Re: DFS Used V/S Non DFS Used

Thanks Suresh - still not clear

Say the dfs.datanode.du.reserved is not set (default seems 0). The non DFS 
Used reported number is non-zero. What does this means ? What is being 
referred as temp files ? and how can they encroach in the example of 
/disk1/datanode, /disk2/datanode etc.

Thanks,

On Fri, Oct 10, 2014 at 2:29 PM, Suresh Srinivas 
sur...@hortonworks.commailto:sur...@hortonworks.com wrote:
Here is the information from - 
https://issues.apache.org/jira/browse/HADOOP-4430?focusedCommentId=12640259page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12640259
Here are the definition of data reported on the Web UI:
Configured Capacity: Disk space corresponding to all the data directories - 
Reserved space as defined by dfs.datanode.du.reserved
DFS Used: Space used by DFS
Non DFS Used: 0 if the temporary files do not exceed reserved space. Otherwise 
this is the size by which temporary files exceed the reserved space and 
encroach into the DFS configured space.
DFS Remaining: (Configured Capacity - DFS Used - Non DFS Used)
DFS Used %: (DFS Used / Configured Capacity) * 100
DFS Remaining % = (DFS Remaining / Configured Capacity) * 100

On Fri, Oct 10, 2014 at 2:21 PM, Manoj Samel 
manojsamelt...@gmail.commailto:manojsamelt...@gmail.com wrote:
Hi,

Not clear how this computation is done

For sake of discussion Say the machine with data node has two disks /disk1 and 
/disk2. And each of these disk has a directory for data node and a directory 
for non-datanode usage.

/disk1/datanode
/disk1/non-datanode
/disk2/datanode
/disk2/non-datanode

The dfs.datanode.data.dir says /disk1/datanode,/disk2/datanode.

With this, what does the DFS and NonDFS indicates? Does it indicates 
SUM(/disk*/datanode)  SUM(/disk*/non-datanode) etc. resp. ?

Thanks,





--
http://hortonworks.com/download/

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

RE: Failed to active namenode when config HA

[Brahma Reddy Battula]

You need to start the ZKFC process which will monitor and manage  the state of 
namenode.





Automatic failover adds two new components to an HDFS deployment: a ZooKeeper 
quorum, and the ZKFailoverController process (abbreviated as ZKFC).

Apache ZooKeeper is a highly available service for maintaining small amounts of 
coordination data, notifying clients of changes in that data, and monitoring 
clients for failures. The implementation of automatic HDFS failover relies on 
ZooKeeper for the following things:

  *   Failure detection - each of the NameNode machines in the cluster 
maintains a persistent session in ZooKeeper. If the machine crashes, the 
ZooKeeper session will expire, notifying the other NameNode that a failover 
should be triggered.
  *   Active NameNode election - ZooKeeper provides a simple mechanism to 
exclusively elect a node as active. If the current active NameNode crashes, 
another node may take a special exclusive lock in ZooKeeper indicating that it 
should become the next active.

The ZKFailoverController (ZKFC) is a new component which is a ZooKeeper client 
which also monitors and manages the state of the NameNode. Each of the machines 
which runs a NameNode also runs a ZKFC, and that ZKFC is responsible for:

  *   Health monitoring - the ZKFC pings its local NameNode on a periodic basis 
with a health-check command. So long as the NameNode responds in a timely 
fashion with a healthy status, the ZKFC considers the node healthy. If the node 
has crashed, frozen, or otherwise entered an unhealthy state, the health 
monitor will mark it as unhealthy.
  *   ZooKeeper session management - when the local NameNode is healthy, the 
ZKFC holds a session open in ZooKeeper. If the local NameNode is active, it 
also holds a special lock znode. This lock uses ZooKeeper's support for 
ephemeral nodes; if the session expires, the lock node will be automatically 
deleted.
  *   ZooKeeper-based election - if the local NameNode is healthy, and the ZKFC 
sees that no other node currently holds the lock znode, it will itself try to 
acquire the lock. If it succeeds, then it has won the election, and is 
responsible for running a failover to make its local NameNode active. The 
failover process is similar to the manual failover described above: first, the 
previous active is fenced if necessary, and then the local NameNode transitions 
to active state.



Please go through following link for more details..


http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html




Thanks  Regards



Brahma Reddy Battula




From: 清如许 [475053...@qq.com]
Sent: Tuesday, September 30, 2014 8:54 AM
To: user
Subject: Re: Failed to active namenode when config HA

Hi, Matt

Thank you very much for your response!

There were some mistakes in my description as i wrote this mail in a hurry. I 
put those properties is in hdfs-site.xml not core-site.xml.

There are four name nodes because i also using HDFS federation, so there are 
two nameservices in porperty
namedfs.nameservices/name
and each nameservice will have two namenodes.

If i configure only HA (only one nameservice), everything is ok, and HAAdmin 
can determine the namenodes nn1, nn3.

But if i configure two nameservice and set namenodes nn1,nn3 for nameservice1 
and nn2,nn4 for nameservices2. I can start these namenodes successfully and the 
namenodes are all in standby state at th beginning. But if i want to change one 
namenode to active state, use command
hdfs haadmin -transitionToActive nn1
HAAdmin throw exception as it cannot determine the four 
namenodes(nn1,nn2,nn3,nn4) at all.

Do you used to configure HAFederation and know what may cause these problem?

Thanks,
Lucy

-- Original --
From:  Matt Narrell;matt.narr...@gmail.com;
Send time: Monday, Sep 29, 2014 6:28 AM
To: useruser@hadoop.apache.org;
Subject:  Re: Failed to active namenode when config HA

I’m pretty sure HDFS HA is relegated to two name nodes (not four), designated 
active and standby.  Secondly, I believe these properties should be in 
hdfs-site.xml NOT core-site.xml.

Furthermore, I think your HDFS nameservices are misconfigured.  Consider the 
following:

?xml version=1.0?
configuration
  property
namedfs.replication/name
value3/value
  /property
  property
namedfs.namenode.name.dir/name
valuefile:/var/data/hadoop/hdfs/nn/value
  /property
  property
namedfs.datanode.data.dir/name
valuefile:/var/data/hadoop/hdfs/dn/value
  /property

property
  namedfs.ha.automatic-failover.enabled/name
  valuetrue/value
/property
property
  namedfs.nameservices/name
  valuehdfs-cluster/value
/property

property
  namedfs.ha.namenodes.hdfs-cluster/name
  valuenn1,nn2/value
/property
  property
namedfs.namenode.rpc-address.hdfs-cluster.nn1/name
valuenamenode1:8020/value
  /property
  property

RE: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN

[Brahma Reddy Battula]

ZKFC LOG:

By Default , it will be under HADOOP_HOME/logs/hadoop_**zkfc.log

Same can be confirmed by using the following commands(to get the log location)

jinfo 7370 | grep -i hadoop.log.dir

ps -eaf | grep -i DFSZKFailoverController | grep -i hadoop.log.dir

WEB Console :

And Default port for NameNode web console is 50070. you can check value of 
dfs.namenode.http-address in hdfs-site.xml..

Default values, you can check from the following link..

http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml





Thanks  Regards

Brahma Reddy Battula






From: arthur.hk.c...@gmail.com [arthur.hk.c...@gmail.com]
Sent: Monday, August 04, 2014 6:07 PM
To: user@hadoop.apache.org
Cc: arthur.hk.c...@gmail.com
Subject: Re: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to 
trigger a roll of the active NN

Hi,

Thanks for your reply.
It was about StandBy Namenode not promoted to Active.
Can you please advise what the path of ZKFC logs?

Similar to Namenode status web page, a Cluster Web Console is added in 
federation to monitor the federated cluster at 
http://any_nn_host:port/dfsclusterhealth.jsp. Any Namenode in the cluster can 
be used to access this web page”
What is the default port for the cluster console? I tried 8088 but no luck.

Please advise.

Regards
Arthur




On 4 Aug, 2014, at 7:22 pm, Brahma Reddy Battula 
brahmareddy.batt...@huawei.commailto:brahmareddy.batt...@huawei.com wrote:

HI,


DO you mean Active Namenode which is killed is not transition to STANDBY..?

 Here Namenode will not start as standby if you kill..Again you need to 
 start manually.

  Automatic failover means when over Active goes down Standy Node will 
transition to Active automatically..it's not like starting killed process and 
making the Active(which is standby.)

Please refer the following doc for same ..( Section : Verifying automatic 
failover)

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html

OR

 DO you mean Standby Namenode is not transition to ACTIVE..?

 Please check ZKFC logs,, Mostly this might not happen from the logs you 
 pasted


Thanks  Regards



Brahma Reddy Battula




From: arthur.hk.c...@gmail.commailto:arthur.hk.c...@gmail.com 
[arthur.hk.c...@gmail.commailto:arthur.hk.c...@gmail.com]
Sent: Monday, August 04, 2014 4:38 PM
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
Cc: arthur.hk.c...@gmail.com
Subject: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a 
roll of the active NN

Hi,

I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying 
automatic failover, after killing the process of namenode from Active one, the 
name node was not failover to standby node,

Please advise
Regards
Arthur


2014-08-04 18:54:40,453 WARN 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
roll of the active NN
java.net.ConnectException: Call From standbynode  to  activenode:8020 failed on 
connection exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1414)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599