Re: Hadoop-MapReduce

Hi, I am using hadoop 0.20 version In that while exceuting the XmlInformat class I am getting the error as "Error: Found Class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was excepted,." Please suggest to fix the error. Thanks in advance. Ranjini On Wed, Dec 11, 2013 at 12

what is MPP,HAWQ,and the relation between them and hadoop?

hi,maillist: ATT

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

yes, hadoop.tmp.dir is both local and hdfs . 2013/12/17 Raviteja Chirala > If I am not wrong, hadoop.tmp.dir is both local and hdfs. Whatever mount > dir, create same in hdfs. > ― > Sent from Mailbox for iPad > > > On Mon, Dec 16, 2013 at 5:05 PM, Tao Xiao wrot

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

If I am not wrong, hadoop.tmp.dir is both local and hdfs. Whatever mount dir, create same in hdfs.  — Sent from Mailbox for iPad On Mon, Dec 16, 2013 at 5:05 PM, Tao Xiao wrote: > Thanks very much, I suppose I know what I should do with > On Mon, Dec 16, 2013 at 5:27 PM, Vinayakumar B > wrote

Run the ??Selenium Webdriver?? in the mapreduce frame of Hadoop, freezed in the map step.

Hello, I'm sorry to disturb you. Few days ago, I decided to let selenium webdriver(a third-party package) run in mapreduce frame of hadoop. And I met a problem. The map step freeze in "new FirefoxDriver();". The FirefoxDriver class was in the third-party jar named "selenium-server-standalone-2.38.0

Re: HDFS short-circuit reads

Hello John, Short-circuit reads are not on by default. The documentation page you linked to at hadoop.apache.org contains all of the information you need to enable them though. Regarding checking status of short-circuit read programmatically, here are a few thoughts on this: Your application co

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Thanks very much, I suppose I know what I should do with On Mon, Dec 16, 2013 at 5:27 PM, Vinayakumar B wrote: > Hi, > > > > *hadoop.tmp.dir* is not the exact configuration you are looking for > spreading the disk I/O > > > > This is the default base directory ( its single directory not multipl

HDFS short-circuit reads

Our YARN application would benefit from maximal bandwidth on HDFS reads. But I'm unclear on how short-circuit reads are enabled. Are they on by default? Can our application check programmatically to see if the short-circuit read is enabled? Thanks, john RE: https://hadoop.apache.org/docs/current/

Re: Getting following error in JT logs while running MR jobs

Looks like thats task tracker issue. Its not able to update JT in time. From Job tracker UI, go to logs and check on that particular failed task in that particular task tracker and check logs. On Dec 16, 2013, at 2:17 AM, Viswanathan J wrote: > Hi, > > I'm getting the following error frequ

How to set the environment variables in Hadoop 2.2.0

The hadoop document below suggests that the following variables be set inorder for Hadoop to prioritize the client jars over the Hadoop jars , however , I am not sure how to set them can someone please tell me how to set these . *HADOOP_USER_CLASSPATH_FIRST=true* and *HADOOP_CLASSPATH* =...:hadoo

[no subject]

Is it possible to access the YARN webpages in a text browser?

Is it possible to access the YARN webpages in a text browser?

Is it possible to access the YARN webpages in a text browser? -- Thanks,

org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException

I am trying to run an example, but I get a PriviledgedActionException. How should I configure the hdfs-site.xml? 2013-12-16 13:43:22,892 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ec2-user (auth:SIMPLE) cause:java.io.IOException: File /tmp/l ogs/ec2-

Re: pipes on hadoop 2.2.0 crashes

You should navigate to the ResourceManager UI following the link and see what is happening on the ResourceManager as well as the application-master. Check if any nodes are active first. Then look at ResourceManager and NodeManager logs. +Vinod On Dec 16, 2013, at 10:29 AM, Mauro Del Rio wrote:

Re: pipes on hadoop 2.2.0 crashes

I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a simple wordcount in c++ with pipes, this time there was no exception, but the job didn't finish. This is the output on the shell where I launched pipes: mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc -input t

Re: Pluggable distribute cache impl

If the files are already on a NFS mount, you don't need to spread files around distributed-cache? BTW, running jobs on NFS mounts isn't going to scale after a while. Thanks, +Vinod On Dec 15, 2013, at 1:15 PM, Jay Vyas wrote: > are there any ways to plug in an alternate distributed cache imp

RE: Yarn -- one of the daemons getting killed

If it is not killed by OOM killer, maybe the JVM just did a core dump due to whatever reason. Search for core dump of process in the /var/log/messages, or core dump file in your system. From: stuck...@umd.edu To: user@hadoop.apache.org; user@hadoop.apache.org Subject: Re: Yarn -- one of the daemo

Re: Getting following error in JT logs while running MR jobs

Hadoop 1.2.1, also I gave permissions to job users. On Dec 16, 2013 8:54 PM, "Mohammad Tariq" wrote: > Which version?? > > Warm Regards, > Tariq > cloudfront.blogspot.com > > > On Mon, Dec 16, 2013 at 4:26 PM, shashwat shriparv < > dwivedishash...@gmail.com> wrote: > >> Does your job fails? check

Re: Yarn -- one of the daemons getting killed

What if you open the daemons in a "screen" session rather than running them in the background -- for example, run "yarn resourcemanager". Then you can see exactly when they terminate, and hopefully why. From: Krishna Kishore Bonagiri Sent: Monday, December 16, 2013 6:20 AM To: user@hadoop.apache

Re: Debug Hadoop Junit Test in Eclipse

In that case i guess you will have to statically trace the code your self. On Mon, Dec 16, 2013 at 10:32 AM, Karim Awara wrote: > > Useful post, however, I am not trying to debug mapreduce programs with > its associated VMs. I want to modify HDFS source code on how it uploads > files. So I am

Re: issue when using HDFS

It is possible that your datanode daemon has not started yet. Logon to the datanode and check if the daemon is running by issue a jps command. Another possible reason is that your namenode can not communicate with datanode. Try ping datanode from the namenode. The log files are supposed in HADOOP

Re: Debug Hadoop Junit Test in Eclipse

Useful post, however, I am not trying to debug mapreduce programs with its associated VMs. I want to modify HDFS source code on how it uploads files. So I am only looking to trace fs commands through the DFS shell. I believe this should be require less work in debugging than actually going to mapr

Re: How does mapreduce job determine the compress codec

Thanks Azurry. That was exactly the thing I want to know. On Sun, Dec 15, 2013 at 7:53 PM, Azuryy Yu wrote: > Hi Jiayu, > For the Sequence file as an input, CompressCodec class was serialized in > the file header, then Sequence Filereader will know the compression algo. > thanks. > > > > > On M

Re: Getting following error in JT logs while running MR jobs

Which version?? Warm Regards, Tariq cloudfront.blogspot.com On Mon, Dec 16, 2013 at 4:26 PM, shashwat shriparv < dwivedishash...@gmail.com> wrote: > Does your job fails? check for the error in log or in the jobtracker web > ui by clicked on failed task you will get a closer error message > > >

Re: Debug Hadoop Junit Test in Eclipse

Excellent question. its not trivial to debug a distributed app in eclipse, but it is totally doable using javaagents . We've written it up here: http://jayunit100.blogspot.com/2013/07/deep-dive-into-hadoop-with-bigtop-and.html FYI cc Brad childs (https://github.com/childsb) at red hat has help

Re: Yarn -- one of the daemons getting killed

Hi Vinay, In the out files I could see nothing other than the output of ulimit -all . Do I need to enable any other kind of logging to get more information? Thanks, Kishore On Mon, Dec 16, 2013 at 5:41 PM, Vinayakumar B wrote: > Hi Krishna, > > > > Please check the out files as well for dae

Debug Hadoop Junit Test in Eclipse

Hi, I want to trace how a file upload (-put) happens in hadoop. So Im junit testing TestDFSShell.java. When I try to debug the test, It fails due to test timed out exception. I believe this is because I am trying to stop one thread while the rest are working. I have changed the breakpoint property

RE: Yarn -- one of the daemons getting killed

Hi Krishna, Please check the out files as well for daemons. You may find something. Cheers, Vinayakumar B From: Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com] Sent: 16 December 2013 16:50 To: user@hadoop.apache.org Subject: Re: Yarn -- one of the daemons getting killed Hi Vinod, Y

Re: Yarn -- one of the daemons getting killed

Hi Vinod, Yes, I am running on Linux. I was actually searching for a corresponding message in /var/log/messages to confirm that OOM killed my daemons, but could not find any corresponding messages there! According to the following link, it looks like if it is a memory issue, I should see a mess

Re: issue when using HDFS

where should I find this logs? I think the problem mainly on slaves,where should I find the logs? 2013/12/16 shashwat shriparv > Had your upgrade finished successfully?? check if datanode is able to > connect to namenode, check datanode logs and please attach some log here if > you are getting

RE: println ststements on hadoop

Hi Shashwat , Go to jobtracker GUI click on running job, then click on 'map'. Select any task running. select Task Logs (ALL). Here is the place all println in your maptask will displays. Thanks & Regards, Job M Thomas| Platform & Technology From: shashwa

Re: Getting following error in JT logs while running MR jobs

Does your job fails? check for the error in log or in the jobtracker web ui by clicked on failed task you will get a closer error message *Warm Regards_**∞_* *Shashwat Shriparv* Big-Data Engineer(HPC) [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]

Re: Getting following error in JT logs while running MR jobs

Check if you are having correct permission for the user through which you are running the job. * Warm Regards_**∞_* *Shashwat Shriparv* Big-Data Engineer(HPC) [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9][image: https

Re: println ststements on hadoop

It will not be displayed in consol as the job is submitted to hadoop, and once it is submitted you dont have control through terminal, meanwhile you can execute hadoop job command giving job id here to get information about the submitted job, how did you submitted the job can you let us know? *W

Re: issue when using HDFS

Had your upgrade finished successfully?? check if datanode is able to connect to namenode, check datanode logs and please attach some log here if you are getting any error in if data node is running. *Warm Regards_**∞_* *Shashwat Shriparv* Big-Data Engineer(HPC) [image: http://www.linkedin.com/

Re: issue when using HDFS

Now the datanode is not working [image: 内嵌图片 1] 2013/12/16 Geelong Yao > it is the namenode's problem. > How can I fix this problem? > > > > 2013/12/16 Shekhar Sharma > >> Seems like DataNode is not running or went dead >> Regards, >> Som Shekhar Sharma >> +91-8197243810 >> >> >> On Mon, Dec 1

Re: MiniDFSCluster setup

You were right on both of them! It is now working. Thanks a lot. -- Best Regards, Karim Ahmed Awara On Mon, Dec 16, 2013 at 4:53 AM, Vinayakumar B wrote: > Hi Kareem, > > Hi See that issue you are facing is > https://issues.apache.org/jira/browse/HADOOP-9908, > > > > It expectes common-versio

println ststements on hadoop

When i run an MR job it completed successfully in JobTracker web UI. Where can i see the println statements which i hv mentioned in map and reduce block? It is not displayed in my console. -- *Thanks & Regards* Unmesha Sreeveni U.B *Junior Developer*

Getting following error in JT logs while running MR jobs

Hi, I'm getting the following error frequently while running MR jobs. ERROR org.apache.hadoop.mapred.TaskStatus: Trying to set finish time for task attempt_201312040159_126927_m_00_0 when no start time is set, stackTrace is : java.lang.Exception at org.apache.hadoop.mapred.TaskStatus.

Re: issue when using HDFS

it is the namenode's problem. How can I fix this problem? 2013/12/16 Shekhar Sharma > Seems like DataNode is not running or went dead > Regards, > Som Shekhar Sharma > +91-8197243810 > > > On Mon, Dec 16, 2013 at 1:40 PM, Geelong Yao wrote: > > Hi Everyone > > > > After I upgrade the hadoop t

RE: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Hi, hadoop.tmp.dir is not the exact configuration you are looking for spreading the disk I/O This is the default base directory ( its single directory not multiple) used in case you didn’t configure your own directories for processes such as NameNode, DataNode and NodeManager. Exact configura

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Hi Tao, No, you need to set mapred.local.dir in the mapred-site.xml with comma separated list of path to spread I/O . On Mon, Dec 16, 2013 at 5:11 PM, Tao Xiao wrote: > Thanks. > > In order to spread I/O among multiple disks, should I assign a > comma-separated list of directories which are l

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Thanks. In order to spread I/O among multiple disks, should I assign a comma-separated list of directories which are located on different disks to "hadoop.tmp.dir"？ for example, hadoop.tmp.dir /mnt/disk1/hadoop_tmp_dir,/mnt/disk2/hadoop_tmp_dir,/mnt/disk3/hadoop_tmp_dir 2013/12/16 S

Re: issue when using HDFS

Seems like DataNode is not running or went dead Regards, Som Shekhar Sharma +91-8197243810 On Mon, Dec 16, 2013 at 1:40 PM, Geelong Yao wrote: > Hi Everyone > > After I upgrade the hadoop to CDH 4.2.0 Hadoop 2.0.0,I try to running some > test > When I try to upload file to HDFS,error comes: > >

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

hadoop.tmp.dir is a directory created on local file system For example if you have set hadoop.tmp.dir property to /home/training/hadoop This directory will be created when you format the namenode by running the command hadoop namenode -format When you open this folder you will see two subfolder

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Hi, Make sure to also set mapred.local.dir to the same set of output directories, this is were the intermediate key-value pairs are stored! Regards, Dieter 2013/12/16 Tao Xiao > I have ten disks per node,and I don't know what value I should set to > "hadoop.tmp.dir". Some said this property r

issue when using HDFS

Hi Everyone After I upgrade the hadoop to CDH 4.2.0 Hadoop 2.0.0,I try to running some test When I try to upload file to HDFS,error comes: node32:/software/hadoop-2.0.0-cdh4.2.0 # hadoop dfs -put /public/data/carinput1G_BK carinput1G DEPRECATED: Use of this script to execute hdfs command is dep

Re: pipes on hadoop 2.2.0 crashes

Ok, I will try. Il 13/dic/2013 19:23 "Vinod Kumar Vavilapalli" ha scritto: > > Could it just be LocalJobRunner? Can you try it on a cluster? We've tested > pipes on clusters, so will be surprised if it doesn't work there. > > Thanks, > +Vinod > > On Dec 13, 2013, at 7:44 AM, Mauro Del Rio wrote: