Re: Hadoop 2.6.0 Error

please also set correct JAVA_HOME in hadoop-env.sh. On Wed, Mar 25, 2015 at 1:53 PM, Anand Murali wrote: > Dear All: > > Request help/advise as I am unable to start Hadoop. Performed follow steps > in Ubuntu 14.10 > > 1. ssh localhost > 2. Did following exports in user defined hadoop.sh and ran

Re: Something about the Snappy Compression Tool

Hi, You should compile Hadoop source code with -Drequire.snappy, such as: mvn -DskipTests -Pnative,dist -Dtar,require.snappy package but you also need to install snappy before your compiling.. On Wed, Mar 25, 2015 at 9:39 AM, 王鹏飞 wrote: > How to recompiling library?Cause my hadoop was built

Re: Passing an array of float type to mapper class

please take a look at LongWritable.java, you need to implement a similiar class. On Wed, Mar 18, 2015 at 3:43 PM, Vandana kumari wrote: > Hello all > I want to pass an array of float type to mapper class. How to do that. Any > help? > > -- > Thanks and regards > Vandana kumari >

Re: Unable to see application in http://localhost:8088/cluster/apps

Did you start JHS(job history server)? On Wed, Mar 18, 2015 at 1:51 AM, Ravi Prakash wrote: > Perhaps yarn.resourcemanager.max-completed-applications ? > > > > > On Tuesday, March 17, 2015 10:02 AM, hitarth trivedi < > t.hita...@gmail.com> wrote: > > > Hi, > > When I submit a job to yarn Reso

Re: Prune out data to a specific reduce task

Hi, Can you set only one reduce task? why did you want set up two reudce tasks and only one work? On Mon, Mar 16, 2015 at 9:04 AM, Drake민영근 wrote: > Hi, > > If you write custom partitioner, just call them to confrim the key match > with which partition. > > You can get the number of reduer from

Re: Snappy Configuration in Hadoop2.5.2

ARY_PATH. > But they didn't work. > > > ------ Original -- > *From: * "Azuryy Yu";; > *Send time:* Monday, Mar 16, 2015 2:43 PM > *To:* "user@hadoop.apache.org"; > *Subject: * Re: Snappy Configuration in Hadoop2.5.2 > > Hi, > please : > run &

Re: Snappy Configuration in Hadoop2.5.2

Hi, please : run "cp -a /opt/snappy/lib/libsnappy.* /opt/hadoop/hadoophome/lib/native" on each datanode. also you need to install Snappy on each datanode firstly. On Sat, Mar 7, 2015 at 6:57 PM, donhoff_h <165612...@qq.com> wrote: > Hi, experts. > > I meet the following problem when configu

Re: File is not written on HDFS after running libhdfs C API

and yarn-site.xml are those on apache website. A > *jps > *command shows: > > 11257 NodeManager > 11129 ResourceManager > 11815 Jps > 10620 NameNode > 10966 SecondaryNameNode > > On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu wrote: > >> Can you share your

Re: File is not written on HDFS after running libhdfs C API

hdfs://localhost:9000 > > > > On Thu, Mar 5, 2015 at 10:32 AM, Alexandru Calin < > alexandrucali...@gmail.com> wrote: > >> No change at all, I've added them at the start and end of the CLASSPATH, >> either way it still writes the file on the local fs.

Re: File is not written on HDFS after running libhdfs C API

, Mar 5, 2015 at 10:22 AM, Azuryy Yu wrote: > >> Yes, you should do it:) >> >> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin < >> alexandrucali...@gmail.com> wrote: >> >>> Wow, you are so right! it's on the local filesystem! Do I have to

Re: File is not written on HDFS after running libhdfs C API

hadoop/etc/hadoop/core-site.xml > ? > > On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu wrote: > >> you need to include core-site.xml as well. and I think you can find >> '/tmp/testfile.txt' on your local disk, instead of HDFS. >> >> if so, My guess i

Re: File is not written on HDFS after running libhdfs C API

you need to include core-site.xml as well. and I think you can find '/tmp/testfile.txt' on your local disk, instead of HDFS. if so, My guess is right. because you don't include core-site.xml, then your Filesystem schema is file:// by default, not hdfs://. On Thu, Mar 5, 2015 at 3:52 PM, Alexa

Re: Need advice about OLAP on Hadoop

Hi VK, I have a similar requirement. we need a real time data analysis platform. Actually, you don't pay more attention on the Spark or Apache Drill, because data for LOAP cubes was calculated before cube build. you just consider two questions: 1) how to calculate the data for cube quickly? 2)w

Re: Can't list files in a federation of HDFS

For HDFS federation, data share all datanodes, but namespace is separate, so did you write some data on hadoop-coc-2 namespace? you don't need to login hadoop-coc-2 then write data, just config a new client, which connect to hadoop-coc-2 for write. On Tue, Mar 3, 2015 at 6:20 PM, xeonmailingli

Re: The Activities of Apache Hadoop Community

That's good to know, On Tue, Mar 3, 2015 at 8:12 PM, Akira AJISAKA wrote: > Hi all, > > One year after the previous post, we collected and analyzed > JIRA tickets again to investigate the activities of Apache Hadoop > community in 2014. > > http://ajisakaa.blogspot.com/2015/02/the-activities-of-

Re: Hadoop svn cannot connected.

Oh, thanks. On Fri, Feb 6, 2015 at 4:58 PM, Andre Kelpe wrote: > Hadoop has moved to git: https://wiki.apache.org/hadoop/GitAndHadoop > > -- André > > On Fri, Feb 6, 2015 at 9:13 AM, Azuryy Yu wrote: > >> Hi, >> >> http://svn.apache.org/viewcvs.cgi/hadoop/co

Hadoop svn cannot connected.

Hi, http://svn.apache.org/viewcvs.cgi/hadoop/common/trunk/ I cannot open this URL. does that anybody can access it? another, I cannot "svn up" the new release in branch-2. It always stay in Aug 2014.

Re: How I list files in HDFS?

Hi, You cannot use new File("...") as parameter, which should be new Path("/outputmp") On Fri, Feb 6, 2015 at 3:51 AM, Ravi Prakash wrote: > Hi Xeon! > > Can you try using the FileContext or FileSystem API? > > HTH > Ravi > > > On Thursday, February 5, 2015 8:13 AM, xeonmailinglist < > xe

Re: Which [open-souce] SQL engine atop Hadoop?

please look at: http://mail-archives.apache.org/mod_mbox/tajo-user/201502.mbox/browser On Tue, Jan 27, 2015 at 5:13 PM, Daniel Haviv wrote: > Can you elaborate on why you prefer Tajo? > > Daniel > > On 27 בינו׳ 2015, at 10:35, Azuryy Yu wrote: > > You almost list all

Re: Compilation failing for AuthenticatedURL (no suitable constructor found) Hadoop 2.6.0

Can you build Hadoop on Linux? or can you check out the whole code from branch-2 again, then build it? On Thu, Feb 5, 2015 at 10:21 AM, Venkat Ramakrishnan < venkat.archit...@gmail.com> wrote: > I need to compile Hadoop, and I am not able to > figure out why this error is occuring. > > Can someon

Re: How to rolling upgrade??

please refer to : http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html On Tue, Feb 3, 2015 at 5:38 PM, Mr.J wrote: > my cluster A, and cluster B. To upgrade to version 2.6 > > In what order should I upgrade? > > Journalnode 1 >> Journalnode 2 > Journalno

Re: Which [open-souce] SQL engine atop Hadoop?

You almost list all open sourced MPP real time SQL-ON-Hadoop. I prefer Tajo, which was relased by 0.9.0 recently, and still working in progress for 1.0 On Mon, Jan 26, 2015 at 10:19 PM, Samuel Marks wrote: > Since Hadoop came out, there have been various > commercial

Re: MapReduce job is not picking up appended data.

Are you sure you can 'cat' the lastest batch of the data on HDFS? for Flume, the data is available only after file rolled, because Flume only call FileSystem.close() during file rolling. On Mon, Jan 26, 2015 at 8:17 PM, Uthayan Suthakar < uthayan.sutha...@gmail.com> wrote: > I have a Flume which

Re: Time until a datanode is marked as dead

Hi Frank, can you file an issue to add this configuration to the hdfs-default.xml? On Mon, Jan 26, 2015 at 5:39 PM, Frank Lanitz wrote: > Hi, > > Am 23.01.2015 um 19:23 schrieb Chris Nauroth: > > The time period for determining if a datanode is dead is calculated as a > > function of a few diff

Re: Multiple separate Hadoop clusters on same physical machines

Hi, I think the best way is deploy HDFS federation with Hadoop 2.x. On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer < harun.za...@tubitak.gov.tr> wrote: > Hi everyone, > > We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, > pig, hive etc.) on 7 physical servers. We wan

Re: Failure to write to HDFS in MapReduce job

please check your network issue. generally this was casued by unstable network device. On Wed, Nov 5, 2014 at 5:55 PM, Hayden Marchant wrote: > > I have a MapReduce job running on Hadoop 2.0.0, and on some 'heavy' jobs, > I am seeing the following errors in the reducer. > > > 2014-11-04 13:30:57

Re: HDFS - Consolidate 2 small volumes into 1 large volume

yes, you can. stop the cluster, change your hdfs-site.xml on your datanode. (dfs.datanode.dir) to the large volume. copy two small data volumes to the large volumes, which was configured on above. start cluster. then you are done. On Tue, Oct 21, 2014 at 9:57 PM, Brian C. Huffman < bhuff...@etin

Re: issue about submit job to local ,not to cluster

please check your mapred-site.xml is available under conf. On Tue, Oct 21, 2014 at 2:47 PM, ch huang wrote: > hi,maillist: >my cluster move from one IDC to another IDC ,when all done ,i run > job ,and find the job run on local box not on cluster ,why? it is normal on > old IDC! >

Re: hadoop 2.4 using Protobuf - How does downgrade back to 2.3 works ?

after upgrade, there are current an previous subdirs in the namenode metadata directory. so during rollback, just delete current directory and rename previous to current again. On Tue, Oct 21, 2014 at 9:27 AM, Manoj Samel wrote: > Is the pre-upgrade metadata also kept updated with any changes on

Re: hadoop 2.4 using Protobuf - How does downgrade back to 2.3 works ?

just stop your cluster, then start your HDFS with '-rollback'. but it's only if you don't finalize HDFS upgrade using command line. On Fri, Oct 17, 2014 at 8:15 AM, Manoj Samel wrote: > Hadoop 2.4.0 mentions that FSImage is stored using protobuf. So upgrade > from 2.3.0 to 2.4 would work since 2

Re: how to copy data between two hdfs cluster fastly?

Did you specified how many map tasks? On Fri, Oct 17, 2014 at 4:58 PM, ch huang wrote: > hi,maillist: > i now use distcp to migrate data from CDH4.4 to CDH5.1 , i > find when copy small file,it very good, but when transfer big data ,it very > slow ,any good method recommand? thanks

Re: C++ development framework under Hadoop

yes. it always supports hadoop pipe in v2. On Wed, Oct 15, 2014 at 11:33 AM, Y Z wrote: > Thanks, Azuryy! > > I found some examples about Pipes. Is Hadoop Pipes still support in Hadoop > 2.2? > > Sincerely, > Yongan > > On 10/14/2014 11:20 PM, Azuryy Yu wrote: >

Re: C++ development framework under Hadoop

Hadoop streaming is the best option for you. It doesn't has high I/O overhead if you don't add a high I/O in your c++ code. hadoop streaming use buidin MapReduce, it just redirect input/out stream for your c++ application. On Tue, Oct 14, 2014 at 10:33 PM, Y. Z. wrote: > Hi Experts, > > I'm g

Re: Datanode volume full, but not moving to free volume

yes, all blockpool directory names are all same is expected if you don't configure HDFS federation. On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman < bhuff...@etinternational.com> wrote: > Hmmm.. It seems that there's only one block pool per disk. So that > won't help me. :-( > > Also, I se

Re: Datanode volume full, but not moving to free volume

Hi Brian, Did you try set dfs.datanode.fsdataset.volume.choosing.policy to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy then there are some other related options with this policy. you can google it. On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres wrote: > > H

Re: ETL using Hadoop

Hi Moin, Yes, you can replace your ETL using hadoop. but it would be a big change. such as data collection, pre-process, ETL tasks rewrite etc. I don't think there is .NET API in Hadoop. On Wed, Oct 8, 2014 at 2:31 PM, Dattatrya Moin wrote: > > Hi , > > We have our own ETL , but we are plannin

Re: Why I cannot delete all the nameNode metadata?

To make sure your dfs.namenode.name.dir is by default. then, how did you find /user exists? hdfs dfs -ls ? or you checked dfs.datanode.data.dir? if the latter, then don't worry. On Wed, Oct 8, 2014 at 11:56 AM, Tianyin Xu wrote: > Hi, > > I wanna run some experiments on Hadoop which requires a

Re: Datanode disk considerations

I think Brian gave the answer. On Tue, Oct 7, 2014 at 9:13 PM, Brian C. Huffman < bhuff...@etinternational.com> wrote: > What about setting the dfs.datanode.fsdataset.volume.choosing.policy to > org.apache.hadoop.hdfs.server. datanode.fsdataset. > AvailableSpaceVolumeChoosingPolicy? > > Would tha

Re: Failed to rollback from hadoop-2.4.1 to hadoop 2.2.0

yes, this is an issue, I also found that. can you please file an issue? On Sun, Sep 21, 2014 at 12:08 PM, sam liu wrote: > I rollback from 2.4.1 to 2.2.0 and seems 2.2.0 does not has option > upgradeProgress, right? > > I guess it might be a hadoop issue, as I still could not start datanode > a

Re: Realtime sensor's tcpip data to hadoop

Hi Alex, you can try Apache Flume. On Wed, May 7, 2014 at 10:48 AM, Alex Lee wrote: > Sensors' may send tcpip data to server. Each sensor may send tcpip data > like a stream to the server, the quatity of the sensors and the data rate > of the data is high. > > Firstly, how the data from tcpip

Re: Change Yarn webUI time format

resolved, I've filed a JIRA for this. https://issues.apache.org/jira/browse/YARN-1998 On Tue, Apr 29, 2014 at 6:46 PM, Azuryy Yu wrote: > Hi, > how to change the time zone of startTime and finishTime on the yarn web ui? > I cannot find the code, I just found render() retur

Change Yarn webUI time format

Hi, how to change the time zone of startTime and finishTime on the yarn web ui? I cannot find the code, I just found render() returns a long type field, but It shows GMT time zone format. how to change to the local time zone? Thanks.

Re: configure HBase

gt; > On Thu, Apr 24, 2014 at 5:46 PM, Ted Yu wrote: > > Please take a look at https://issues.apache.org/jira/browse/HBASE-6567 > > > > Cheers > > > > On Apr 24, 2014, at 3:33 AM, Azuryy Yu wrote: > > > > Hi, > > > > what's mean of HB

configure HBase

Hi, what's mean of HBASE_REGIONSERVER_MLOCK? I cannot find the doucment for it. there is only "Uncomment and adjust to keep all the Region Server pages mapped to be memory resident" in the hbase-env.sh, can you explain in detail? Thanks for any inputs.

Re: JVM option

Do you want add "-Xmx4g" to your MR tasks? if so, just add it as "mapred.child.java.opts" in the mapred-site.xml On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine wrote: > Quick question. How would I pass the following JVM option to the Hadoop > command line? > > "-Xmx4G" > > hadoop jar > > Thank

Re: hadoop 2.4.0?

Hadoop 2.4.0 doesn't has the known issue now. I think it's a stable release even if it's not in the stable download list. the only one issue I met is that you should upgrade Hive to Hive-0.12.0 after upgrade to 2.4.0 for the API compatible. On Fri, Apr 18, 2014 at 1:07 AM, MrAsanjar . wrote: >

Re: Problem with jobtracker hadoop 1.2

There is because your HDFS has no space left. please check your datanodes are all started. also please check dfs.datanode.du.reserved in hdfs-site.xml to make sure you don't reserve large capacity. On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin wrote: > Did you start datanode service? > > > On T

Re: Hadoop NoClassDefFoundError

Please use: hadoop jar myjob.jar myjob.MyJob input output On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn wrote: > Hello EveryOne: > I am new to hadoop,and i am reading Hadoop in action. > When i tried to run a demo from this book,I got a problem and could not > find answer from the net.

Re: Hadoop NoClassDefFoundError

Please use: hadoop jar myjob.jar myjob.MyJob input output On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn wrote: > Hello EveryOne: > I am new to hadoop,and i am reading Hadoop in action. > When i tried to run a demo from this book,I got a problem and could not > find answer from the net.

Re: Value for yarn.nodemanager.address in configuration file

Hi Krishna, Don't worrry about that, there is no ssh 0.0.0.0 during start NM, it's a local service. On Fri, Apr 4, 2014 at 9:12 AM, Shengjun Xin wrote: > you can use yarn-daemon.sh to start nodemanager without ssh > > > On Thu, Apr 3, 2014 at 10:36 PM, Krishna Kishore Bonagiri < > write2kish...

Re: How to open .gv file for Yarn event model

Thanks Ravi, I am using Graphviz as Jeff said. it's enough to me. On Thu, Apr 3, 2014 at 5:12 AM, Ravi Prakash wrote: > Hi Azuryy! > > You have to use dot to convert it to png > > > On Tuesday, April 1, 2014 6:38 PM, Azuryy Yu wrote: > Hi, > > I compiled Y

Re: Container states trantition questions

> >> same for me. all mapper ends with 143. >> >> I've no idea what it means >> >> >> 2014-04-03 8:45 GMT+08:00 Azuryy Yu : >> >> Hi, >>> >>> Does it normal for each container end with TERMINATED(143) ? >>> The whole M

Fwd: Container states trantition questions

Hi, Does it normal for each container end with TERMINATED(143) ? The whole MR job is successful, but all containers in the map phase end with 143. There are no any useful logs in the NM, AM, Container logs. Another minor question: There are only WARN logs in the stderr: log4j:WARN No appenders c

How to open .gv file for Yarn event model

Hi, I compiled Yarn event model using maven, but how to open .gv file to view it? Thanks.

Re: patch format

There is no special patch format, just using svn diff or git diff to generate the patch, which depeds on your code repository. On Tue, Apr 1, 2014 at 5:14 PM, Avinash Kujur wrote: > > hello everyone, > > Is there any kind of specific format of the patch in which we need to > write the patch fil

Re: patch format

There is no special patch format, just using svn diff or git diff to generate the patch, which depeds on your code repository. On Tue, Apr 1, 2014 at 5:14 PM, Avinash Kujur wrote: > > hello everyone, > > Is there any kind of specific format of the patch in which we need to > write the patch fil

Re: any link for hadoop 1.3.0

There is no release for hadoop 1.3.0, but you can build the source code from: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/ This is candidate branch for 1.3.0 On Mon, Mar 31, 2014 at 5:16 PM, Avinash Kujur wrote: > hi, > > can anyone provide the link for hadoop 1.3.0? > > re

Re: job not shown in job history server

Did you configure mapreduce.jobhistory.address and mapreduce.jobhistory.webapp.address in the mapred-site.xml? On Mon, Mar 31, 2014 at 3:10 PM, Rahul Singh wrote: > Hi, > > I have configured hadoop 2.3 single node setup and ran a sample job > using hadoop command, > hadoop jar > > The job

Re: how to be assignee ?

Hi Avin, You should be added as an sub-project's contributor, then you can be an assignee. so you can find how to be an contributor on the Wiki. On Fri, Mar 28, 2014 at 6:50 PM, Avinash Kujur wrote: > hi, > > how can i be assignee fro a particular issue? > i can't see any option for being assi

Re: HADOOP_MAPRED_HOME not found!

it was defined at hadoop-config.sh On Fri, Mar 28, 2014 at 1:19 PM, divye sheth wrote: > Which version of hadoop are u using? AFAIK the hadoop mapred home is the > directory where hadoop is installed or in other words untarred. > > Thanks > Divye Sheth > On Mar 28, 2014 10:43 AM, "Avinash Kuju

Re: Decommissioning a node takes forever

Hi, which version HDFS you used? On Wed, Mar 26, 2014 at 3:17 PM, Bharath Kumar wrote: > > Hi All, > > I am a novice hadoop user . I tried removing a node from my cluster of 2 > nodes by adding the ip in excludes file and running dfsadmin -refreshNodes > command . But decommissioning takes a ve

Re: namenode could not start

, 2014 at 6:39 PM, Brahma Reddy Battula < >> brahmareddy.batt...@huawei.com> wrote: >> >>> Please format the namenode and then do start-dfs.. >>> >>> >>> >>> >>> >>> *Command for namenode format:* >>> >>> &

Re: namenode could not start

Are you sure /home/hadoop/mydata/hdfs/namenode exists and has a right permission? On Tue, Mar 25, 2014 at 4:51 PM, haihong lu wrote: > Dear all: > > I had a problem that the namenode could not start when i ran start-dfs.sh > . Show the message as below: > > FATAL org.apache.hadoop.hdfs.server.

Re: lot of attempt_local296445216_0001_m_000386_0 dir in NN dir

This is a known issue, but not a bug. one safe way is remove files older than sevral hours under these dirs. On Tue, Mar 25, 2014 at 8:57 AM, ch huang wrote: > hi,maillist: > i find many dirs in > /data/hadoopmapredlocal/taskTracker/hdfs/jobcache/job_local296445216_0001 > ,it is my m

Re: how does hdfs client write a file .

initially the HDFS client caches the file data into a temporary > local file. Application writes are transparently redirected to this > temporary local file. > > > > > 2014-03-24 17:01 GMT+08:00 Azuryy Yu : > > Can you give the URL where you looked at? >> &

Re: how does hdfs client write a file .

Can you give the URL where you looked at? On Mon, Mar 24, 2014 at 4:49 PM, 张建伟 wrote: > hi, > > In the apache site,I find a document said : > > "A client request to create a file does not reach the NameNode > immediately. In fact, initially the HDFS client caches the file data into a > temporar

Re: hadoop source code

Hi, you'd bettern check out from svn, then build with maven: mvn eclipse:eclipse, which will generate eclipse projects to you. On Mon, Mar 24, 2014 at 3:24 PM, Avinash Kujur wrote: > hi, > > i downloaded the hadoop source code from github. after importing those > files in eclipse some of the cl

Re: Hadoop versions

Hi, I do think Hadoop-0.2x is not similar to 2.0. Hadoop version is composed of major-version.minor-version.released such as: Hadoop-2.0.5, major version is 2, 0 is minor versioin. 5 the the fifth release on 2.0 currently, Hadoop-1.2 and Hadoop-2.2 are all stable, but there are big difference bet

Re: Hadoop dfs upgrade fail when HA enabled

It'll be supported in 2.4. please look at here: https://issues.apache.org/jira/browse/HDFS-5138 On Fri, Mar 21, 2014 at 3:46 PM, Meng QingPing wrote: > Hi, > > Hadoop dfs upgrade fail when HA enabled. Can Hadoop add feature to > upgrade dfs based on HA configure automatically ? > > Thanks, >

Re: I am about to lose all my data please help

I don't think this is the case, because there is; hadoop.tmp.dir /home/hadoop/project/hadoop-data On Tue, Mar 18, 2014 at 1:55 PM, Stanley Shi wrote: > one possible reason is that you didn't set the namenode working directory, > by default it's in "/tmp" folder; and the "/tmp" fol

Re: I am about to lose all my data please help

hadoop.tmp.dir /home/hadoop/project/hadoop-data On Tue, Mar 18, 2014 at 2:06 PM, Azuryy Yu wrote: > I don't think this is the case, because there is; > > hadoop.tmp.dir > /home/hadoop/project/hadoop-data > > > > On Tue, Mar 18, 2014 a

Re: The best practice of migrating hadoop 1.0.1 to hadoop 2.2.3

Hi, 1) Is it possible to do an "in-place" migration, while keeping all data in HDFS safely? yes. stop the HDFS firstly, then run "start-dfs.sh -upgrade" 2) If it is yes, is there any doc/guidance to do this? you just want a HDFS upgrade, so I don't think there are some useful doc. 3)

Re: Question on DFS Balancing

at I should take? > Thanks Azuryy. > > > On Wed, Mar 5, 2014 at 2:19 PM, Azuryy Yu wrote: > >> you can write a simple tool to move blocks peer to peer. I had such tool >> before, but I cannot find it now. >> >> background: our cluster is not balanced, load bala

Re: Question on DFS Balancing

d take while > doing this? Data loss is a NO NO for me. > > Thanks > Divye Sheth > > > On Wed, Mar 5, 2014 at 1:28 PM, Azuryy Yu wrote: > >> Hi, >> That probably break something if you apply the patch from 2.x to 0.20.x, >> but it depends on. >> >&g

Re: Question on DFS Balancing

Hi, That probably break something if you apply the patch from 2.x to 0.20.x, but it depends on. AFAIK, Balancer had a major refactor in HDFSv2, so you'd better fix it by yourself based on HDFS-1804. On Wed, Mar 5, 2014 at 3:47 PM, divye sheth wrote: > Thanks Harsh. The jira is fixed in versio

Re: Unable to export hadoop trunk into eclipse

Have you installed protobuf on your computer? https://code.google.com/p/protobuf/downloads/list On Tue, Mar 4, 2014 at 3:08 PM, nagarjuna kanamarlapudi < nagarjuna.kanamarlap...@gmail.com> wrote: > Hi Ted, > > I didn't do that earlier. > > Now , I did it > mvn:eclipse eclipse > and tried impo

Re: hadoop 2.2.0 cluster setup error : could only be replicated to 0 nodes instead of minReplication (=1)

Generally, this is caused by insufficient space. please check the total capacity of your cluster and used, remaining ratio, and check dfs.datanode.du.reserved in the hdfs-site.xml if this value is larger than your remained capacity, then you got this Exception. On Tue, Feb 25, 2014 at 10:35 AM

Re: Having trouble adding external JAR to MapReduce Program

Hi, you cannot add jar like this way. please look at DistributeCache in the Hadoop Java Doc. please call DistributeCache.addArchive() in your main Class before submit the MR job. On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta wrote: > Jonathan, > > > > You have to make sure that the jar is ava

Re: Reduce task hang[EMERGENCE]

actuall these are all logs in the stderr, and stdout is empty On Fri, Jan 3, 2014 at 4:12 PM, Azuryy Yu wrote: > Hi Harsh, > There is only three warnnings in stderr > > *stderr logs* > > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.mapred.Child)

Re: Reduce task hang[EMERGENCE]

Fri, Jan 3, 2014 at 1:59 PM, Harsh J wrote: > Hi again, > > I did not mention the TaskTracker log, but the actual Task JVM's > syslog/stderr/stdout. Do those log *any* problems at all? > > On Fri, Jan 3, 2014 at 11:16 AM, Azuryy Yu wrote: > > Hi Harsh, > > Th

Re: XML to TEXT

Hi, you can use org.apache.hadoop.streaming.StreamInputFormat using map reduce to convert XML to text. such as your xml like this: lll you need to specify stream.recordreader.begin and stream.recordreader.end in the Configuration: Configuration conf = new Configuration(); conf.set("stream.r

Re: Reduce task hang[EMERGENCE]

ng to communicate with the various TaskTrackers > in trying to obtain the data? > > On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu wrote: > > Add addtional: > > > > Our MR version is 1.2.1, not 1.0.4 > > > > There is no useful information in the JT log. > >

Re: Reduce task hang[EMERGENCE]

In detail: 'and these people's job never hanged...' these people's map and reduce tasks never hanged. On Fri, Jan 3, 2014 at 1:46 PM, Azuryy Yu wrote: > Hi Harsh, > Thanks. > > There is no any error logs for attempt_201312201200_34795_r_00_0 in > the

Re: Reduce task hang[EMERGENCE]

Add addtional: Our MR version is 1.2.1, not 1.0.4 There is no useful information in the JT log. On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu wrote: > Hi, > > Our prod cluster met some issues recently, > All map tasks finished successfully, but reduce task hanged. > > but It

Reduce task hang[EMERGENCE]

Hi, Our prod cluster met some issues recently, All map tasks finished successfully, but reduce task hanged. but It's not happened on all TaskTrackers, only sometimes. we used mapred-1.0.4 There is "0.0% reduce > copy >" forever until kill task manually. reduce logs on the TaskTracker: hadoop-ha

Re: get job data in command line in MRv2

Generally, MRv2 indicates Yarn. you can try: yarn application then there are full help lists. On Tue, Dec 31, 2013 at 12:32 PM, 小网客 wrote: > ui or hadoop job command like：hadoop job -list > > -- > - > BestWishes！ > 小网客 > Blog:http://

Re: any suggestions on IIS log storage and analysis?

You can run a mapreduce firstly, Join these data sets into one data set. then analyze the joined dataset. On Mon, Dec 30, 2013 at 3:58 PM, Fengyun RAO wrote: > Hi, > > HDFS splits files into blocks, and mapreduce runs a map task for each > block. However, Fields could be changed in IIS log file

Re: Estimating the time of my hadoop jobs

Hi Kandoi, It depends on: how many cores on each VNode how complicated of your analysis application But I don't think it's normal spent 3hr to process 30GB data even on your *not good* hareware. On Tue, Dec 17, 2013 at 6:39 PM, Kandoi, Nikhil wrote: > Hello everyone, > > > > I am new to Had

Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?

Hi Tao, No, you need to set mapred.local.dir in the mapred-site.xml with comma separated list of path to spread I/O . On Mon, Dec 16, 2013 at 5:11 PM, Tao Xiao wrote: > Thanks. > > In order to spread I/O among multiple disks, should I assign a > comma-separated list of directories which are l

Re: how to create symbolic link in hdfs with c++ code or webhdfs interface?

Why enable test flag under non-test environment? On Mon, Dec 16, 2013 at 11:10 AM, Xiaobin She wrote: > To Vinod, > > thank you for your reply, I found that I got this error message because I > did not add the following property in the hdfs-site.xml file > > 28 > 29 test.SymlinkE

Re: Site-specific dfs.client.local.interfaces setting not respected for Yarn MR container

Jeff, DFSClient don't use copied Configuration from RM. did you add hostname or IP addr in the conf/slaves? if hostname, Can you check /etc/hosts? does there have confilicts? and y On Mon, Dec 16, 2013 at 5:01 AM, Jeff Stuckman wrote: > Thanks for the response. I have the preferIPv4Stack opt

Re: How does mapreduce job determine the compress codec

Hi Jiayu, For the Sequence file as an input, CompressCodec class was serialized in the file header, then Sequence Filereader will know the compression algo. thanks. On Mon, Dec 16, 2013 at 8:28 AM, Jiayu Ji wrote: > Thanks Tao. I know I can tell it is a lzo file based on the magic number. > W

Re: hadoop 2.2 build

Can you check the datanode log to find what error happended? On Sat, Dec 14, 2013 at 9:27 AM, Karim Awara wrote: > Hi, > > I was setting up hadoop 2.2 following > theseinstructions > on a single machine with a tiny

Re: real time analytics on hadoop using spark or storm

spark streaming is a mini job, which can update every 150ms, but storm is long live process. On 2013-12-07 9:12 AM, "Jay Vyas" wrote: > Spark increases performance by using distributed shared memory. > > Storm on the other hand gives you realtime performance by processing data > sets in small ba

Re: issue about read file from HDFS

Ok, so you go for yourself. On Wed, Dec 4, 2013 at 12:15 PM, ch huang wrote: > it seems not a good suggestion,get lot of partition dir and data file will > be a big compact to NN > > > On Wed, Dec 4, 2013 at 12:08 PM, Azuryy Yu wrote: > >> One suggestion is change yo

Re: issue about read file from HDFS

One suggestion is change your hive partition, add a hive partition every five minutes, and your HDFS file also roller every five minutes. On Wed, Dec 4, 2013 at 11:56 AM, ch huang wrote: > hi,mailist: >when HDFS 's file is in appending ,no other reader can get > data from this

Re: Perfect configuration setting

ec 3, 2013 at 5:46 PM, Azuryy Yu wrote: > >> Hi Sandy, >> I think there are some permission limited on your shared URL. maybe only >> Cloudera employees allowed. >> >> >> On Tue, Dec 3, 2013 at 4:26 PM, Sandy Ryza wrote: >> >>> Hi Geel

Re: Perfect configuration setting

Hi Sandy, I think there are some permission limited on your shared URL. maybe only Cloudera employees allowed. On Tue, Dec 3, 2013 at 4:26 PM, Sandy Ryza wrote: > Hi Geelong, > > Check out Todd Lipcon's presentation on tuning MapReduce performance: > http://www.slideshare.net/cloudera/mr-perf >

Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?

Hi Jay, Can you point out a detail about 1.x that job tracker having an ever-expanding hash map. I want fix it. On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas wrote: > version is rewally important here.. > > - If 1.x, then Where (NN , JT , TT ?) > - if 2.x, then where? (AM, NM, ... ?) <-- probably le

Re: Write Access Denied in hdfs

I cannot open you shared URL. but if you write or create file by user A under a HDFS path , which is not created by user A, then it's permission failed. one solution is 'hdfs fs -chown -R A:A your_path' , another is 'hdfs fs -chmod -R 777 your_path' On Tue, Dec 3, 2013 at 5:09 PM, unmesha sreev

  1   2   3   >