Why not remove the block on the disk if the snapshot?

hadoop version 2.4.1. I have tested the snapshot. 1. upload file, /tmp/test.avi 2. create snapshot, /tmp snap1 3. delete file, /tmp/test.avi 4. moved to the Trash, /user/hadoop/.Trash/tmp/test.avi 5. file deleted in trash. 6. but, blocks is not deleted in datanode disk. why?? 7. delete snap

Hadoop InputFormat - Processing large number of small files

Hello, I have a use case wherein i need to process huge set of files stored in HDFS. Those files are non-splittable and they need to be processed as a whole. Here, I have the following question for which I need answers to proceed further in this. 1. I wish to schedule the map process in task tra

Transformation from GZIP to LZO

Hello, Can some one help me how we can convert the file from gzip to Lzo. pls suggest me various ways to do that (MapReduce.Pig.Hive etc) You will be greatly appriciated if you can provide with the code details as well. Thanks in advance for your help on this.

Memory settings in hadoop YARN

Hi We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture. I had few questions on memory used by the jobs : We have following settings for memory. mapred.child.java.opts mapreduce.map.memory.mb mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.a

Why mount point of Name Service can NOT be "/" when enable HDFS federation

Hello, According to CDH documentation (http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.3/Cloudera-Manager-Enterprise-Edition-User-Guide/cmeeug_topic_5_13.html), "/" can NOT be used as mount point. Is there some explicit reason for this? Because of this limitation, if

Starting hadoop on reboot/start

I installed apache hadoop, however I am unable to find any script that I can configure as a service. Does anyone have any steps or scripts that can be reused?

Issue with Hadoop/Kerberos security as client

We are encountering a really strange issue accessing Hadoop securely as a client. We go through the motions of calling setting the security configuration: YarnConfiguration conf = new YarnConfiguration(); conf.set(DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY, nnPrincipal); conf.set(Yarn

Re: mismatch layout version.

BTW, the upgrade works correctly. The issue is only in the log messages. thanks , benoy On Tue, Aug 19, 2014 at 8:05 AM, Benoy Antony wrote: > This is a bug and is handled in > https://issues.apache.org/jira/browse/HDFS-6797 > > thanks and regards, > Benoy > > > On Mon, Aug 18, 2014 at 10:54 P

Re: Hadoop 2.4.1 Snappy Smoke Test failed

Could this be caused by the fact that hadoop no longer ships with 64bit libs? https://issues.apache.org/jira/browse/HADOOP-9911 - André On Tue, Aug 19, 2014 at 5:40 PM, arthur.hk.c...@gmail.com < arthur.hk.c...@gmail.com> wrote: > Hi, > > I am trying Snappy in Hadoop 2.4.1, here are my steps: >

Hadoop 2.4.1 Snappy Smoke Test failed

Hi, I am trying Snappy in Hadoop 2.4.1, here are my steps: (CentOS 64-bit) 1) yum install snappy snappy-devel 2) added the following (core-site.xml) io.compression.codecs org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compre

Re: mismatch layout version.

This is a bug and is handled in https://issues.apache.org/jira/browse/HDFS-6797 thanks and regards, Benoy On Mon, Aug 18, 2014 at 10:54 PM, juil cho wrote: > > > I installed version 2.4.1. > > > > I found the following information in the datanode log file. > > > > But, cluster is healthy. > >

RE: Problems with the Fedarated name node configuration

Hi Liang Yanbo ☺ Thanks for yor reply I could get the federated name node to work – with some changes though – posting for benefit of group Step4) I created ssh cert on my secondary (federated) name node as well ( using ssh-keygen) and copied it on all other data and the existing name nodes

Re: Unsubscribe

The unsubscribe email has to be sent from your email account (to user-unsubscr...@hadoop.apache.org). On Tue, Aug 19, 2014 at 5:03 AM, Haiyang Fu wrote: > Hi, please unsubscribe me too from this list. > > Thank you. > > > On Tue, Aug 19, 2014 at 4:31 PM, tobe wrote: > >> Please have a look at

Re: Hadoop HDFS slow after upgrade vom 0.20 -> to 2.0

Hi Norbert, Can you check the DN daemon's GC activity (you can look for the JvmPauseMonitor messages in logs too, in 4.7)? Is it increased from pre-upgrade, or generally seems high to you? What is your current DN heap size? One of the major changes between 0.20 to 2.0 is the BlockPool ID concept

Unsubscribe

Hi, please unsubscribe me too from this list. Thank you. On Tue, Aug 19, 2014 at 4:31 PM, tobe wrote: > Please have a look at https://hbase.apache.org/mail-lists.html and hit > Unsubscribe . > > > On Tue, Aug 19, 2014 at 4:14 PM, Subroto Sanyal > wrote: > >> Send a mail to user-unsubscr...@ha

RE: Jobtracker could only be replicated to 0 nodes instead of 1

I had the same issue. Can you try disabling firewall from both datanode and resourcemanager using "sudo /etc/init.d/iptables stop"? Regards, Smita From: Sindhu Hosamane [mailto:sindh...@gmail.com] Sent: Saturday, August 16, 2014 2:09 AM To: user@hadoop.apache.org Subject: Re: Jobtracker could on

Re: Unsubscribe

Please have a look at https://hbase.apache.org/mail-lists.html and hit Unsubscribe . On Tue, Aug 19, 2014 at 4:14 PM, Subroto Sanyal wrote: > Send a mail to user-unsubscr...@hadoop.apache.org > > Cheers, > Subroto Sanyal > On 19 Aug 2014, at 13:40, Vasantha Kumar Kannaki Kaliappan < > vaska...@

Re: Unsubscribe

Send a mail to user-unsubscr...@hadoop.apache.org Cheers, Subroto Sanyal On 19 Aug 2014, at 13:40, Vasantha Kumar Kannaki Kaliappan wrote: > unsubscribe signature.asc Description: Message signed with OpenPGP using GPGMail

Unsubscribe

Unsubscribe me please.. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other

Re: Unsubscribe

Hi, please unsubscribe me too from this list. Thank you. 2014-08-19 5:10 GMT-03:00 Vasantha Kumar Kannaki Kaliappan < vaska...@student.liu.se>: > Hi, > > Please unsubscribe me from the list. Thanks a lot for active members. You > guys rock! > > All the best! > > -- > Regards, > > Vasanth >

Unsubscribe

Hi, Please unsubscribe me from the list. Thanks a lot for active members. You guys rock! All the best! -- Regards, Vasanth

Hadoop HDFS slow after upgrade vom 0.20 -> to 2.0

Hi List, we have upgraded Hadoop from our very old version 0.20 to Cloudera 4.7 (hadoop 2.0), we are only using HDFS. After upgrade (no configuration changes), the hdfs seems to bee very slow. It needs more than 2h to copying 40GB(47 files) out of the hdfs, bevor upgrading it was about 1h. W

Re: multi tenancy with cassandra

HDFS is a really a filesystem on top of a traditional filesystem. HDFS is just a folder with files in it (and this may occur on each node in your hadoop cluster). On 19 Aug 2014 08:42, "Adaryl "Bob" Wakefield, MBA" < adaryl.wakefi...@hotmail.com> wrote: > Since Cassandra doesn't store it's data in

multi tenancy with cassandra

Since Cassandra doesn't store it's data in HDFS, is it possible to install Cassandra on the same boxes that are running Hadoop? What I'm really asking is, does HDFS just take over the entire disk or just a segment? Is there something I'm missing in the understanding of HDFS? I've been thinking a