Re: modify HDFS

Hi all, My previous web surfing led me to such steps that I executed successfully. However, my issue is that what version of hadoop this is? (I believe it is hadoop 3.0 since it supports maven build). However, I want to modify a stable release (hadoop 1.1.2, which does not have the maven build sup

HDFS / Federated HDFS - Doubts

Hi All, While trying to understand federated HDFS in detail I had few doubts and listing them down for your help. 1. In case of *HDFS(without HDFS federation)*, the metadata or the data about the blocks belonging to the files in HDFS is maintained in the main memory of the name node or i

Re: modify HDFS

Karim! You should read BUILDING.txt . I usually generate the eclipse files using mvn eclipse:eclipse Then I can import all the projects into eclipse as eclipse projects. This is useful for code navigation and completion etc. however I still build using command line: mvn -Pdist -Dmaven.javado

Re: Uploading a file to HDFS

I've diagramed the hadoop HDFS write path here: http://jayunit100.blogspot.com/2013/04/the-kv-pair-salmon-run-in-mapreduce-hdfs.html On Tue, Oct 1, 2013 at 5:24 PM, Ravi Prakash wrote: > Karim! > > Look at DFSOutputStream.java:DataStreamer > > HTH > Ravi > > > --

Re: Uploading a file to HDFS

Karim! Look at DFSOutputStream.java:DataStreamer HTH Ravi From: Karim Awara To: user Sent: Thursday, September 26, 2013 7:51 AM Subject: Re: Uploading a file to HDFS Thanks for the reply. when the client caches 64KB of data on its own side, do you k

RE: Task status query

Thanks Arun! It is sometimes hard for me to figure out what is built into YARN vs what is done by convention. John From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Monday, September 23, 2013 5:41 PM To: user@hadoop.apache.org Subject: Re: Task status query Yep, typically, the AM should

Re: modify HDFS

Hi, What issue you having. I Wrote it about here , might help you Import it into eclipse as maven project http://jugnu-life.blogspot.com.au/2013/09/build-and-compile-hadoop-from-source.html?m=1 Thanks On 01/10/2013 11:56 PM, "Karim Awara" wrote: > Hi, > > I want to modify the source code of

Re: so the master just died... now what?

I'm using Hadoop 1.1.2, since the latest stable HBase (0.94.x) does not seem to be supported when using Hadoop 2.x (or, at least, was not tested well enough). So what are my best options? Use Hadoop 2.x after all, and follow one of these? http://hadoop.apache.org/docs/current/hadoop-yarn/hado

modify HDFS

Hi, I want to modify the source code of HDFS to my usage, but I can't get any handy sources for development of hdfs on eclipse. (I tried the hdfs developer mailing list, but they are unresponsive). May you guide me? -- Best Regards, Karim Ahmed Awara -- -- This mess

Re: Question on BytesWritable

https://issues.apache.org/jira/browse/HADOOP-6298 On Tue, Oct 1, 2013 at 12:39 AM, Chandra Mohan, Ananda Vel Murugan wrote: > Hi, > > > > I am using Hadoop 1.0.2. I have written a map reduce job. I have a > requirement to process the whole file without splitting. So I have written a > new input f

Building hadoop 2.1.0 beta on windows

I've just downloaded the hadoop 2.1.0 beta source and I'm not having much luck building it (well heaps of test fails) According to BUILDING.txt these are the pre-reqs: * Windows System * JDK 1.6 * Maven 3.0 * Windows SDK or Visual Studio 2010 Professional * ProtocolBuffer 2.4.1+ (for MapReduce a

Re: Cluster config: Mapper:Reducer Task Capapcity

What is the down side of increasing both mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum to same value ? I read on this linkthat: mapred.tasktracker.map.tasks.maximum 1/2 * (cores/node) to 2 * (cores/node)N