RE: one input file per map
Nope, But if the intent is so then there are 2 ways of doing it. 1. Just extend the input format of your choice and override isSplitable() method to return false. 2. Compress your text file using a compression format supported by hadoop (e.g gzip). This will ensure that one map task processes 1 file since compressed files are not split between processes. -Original Message- From: Qiong Zhang [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 01, 2008 9:54 PM To: core-user@hadoop.apache.org Subject: one input file per map Hi, Is there an existing input format/split which supports one input file (e.g. plain text) per map task? Thanks, James
Re: scaling issue, please help
Mori Bellamy wrote: i discovered that some of my code was causing out of bounds exceptions. i cleaned up that code and the map tasks seemed to work. that confuses me -- i'm pretty sure hadoop is resilient to a few map tasks failing (5 out of 13k). before this fix, my remaining 2% of tasks were getting killed. Mori, I am not sure what the confusion is. Hadoop is resilient to few task failures but not by default. The parameter that does it is mapred.max.map.failures.percent and mapred.max.reduce.failures.percent. Every task internally consists of attempts (internally, for the framework). Hadoop allows some attempt failures too. If the number of attempts that failed of a task exceeds the threshold (mapred.map.max.attempts/mapred.reduce.max.attempts : default is 4) then the task is considered failed. If the number of map/reduce task failures exceeds the threshold (mapred.max.map.failures.percent/mapred.max.reduce.failures.percent : default is 0) then the job is considered failed. Amar On Jul 1, 2008, at 10:06 PM, Amar Kamat wrote: Mori Bellamy wrote: hey all, i've got a mapreduce task that works on small (~1G) input. when i try to run the same task on large (~100G) input, i get the following error around when the map tasks are almost done (~98%) 2008-07-01 13:10:59,231 INFO org.apache.hadoop.mapred.ReduceTask: task_200807011005_0005_r_00_0: Got 0 new map-outputs 0 obsolete map-outputs from tasktracker and 0 map-outputs from previous failures 2008-07-01 13:10:59,232 INFO org.apache.hadoop.mapred.ReduceTask: task_200807011005_0005_r_00_0 Got 0 known map output location(s); scheduling... 2008-07-01 13:10:59,232 INFO org.apache.hadoop.mapred.ReduceTask: task_200807011005_0005_r_00_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2008-07-01 13:10:59,232 INFO org.apache.hadoop.mapred.ReduceTask: task_200807011005_0005_r_00_0 Need 1 map output(s) ... ... These are not error messages. The reducers are stuck as not all maps are completed. Mori, could you let us know what is happening to the other 2% maps. Are they getting executed? Are they still pending (waiting to run)? Were they killed/failed? Is there any lost tracker? I'm running the task on a cluster of 5 workers, one DFS master, and one task tracker. What do you mean by 5 workers and 1 task tracker? i'm chaining mapreduce tasks, so i'm using SequenceFileOutput and SequenceFileInput. this error happens before the first link in the chain sucessfully reduces. Can you elaborate this a bit. Are you chaining MR jobs? Amar does anyone have any insight? thanks!
Re: failed map tasks
jerrro wrote: Hello, I was wondering - could someone tell me what are the reasons that I could get failure with certain map tasks on a node? Well, that depends on the kind of errors you are seeing. Could you plz post the logs/error messages? Amar Any idea that comes to mind would work (it does not happen because my program aborts). Also, what could cause with hadoop for a map task to take longer (although data is the same for all maps) to finish than the other tasks? What Hadoop version are you using? What is the average runtime of the maps? How much difference are you seeing? Are all the nodes same in terms of config? Thanks Jerr
RE: topology.script.file.name
This is strange. If you don't mind, pls send the script to me. -Original Message- From: Yunhong Gu1 [mailto:[EMAIL PROTECTED] Sent: Thursday, July 03, 2008 9:49 AM To: core-user@hadoop.apache.org Subject: topology.script.file.name Hello, I have been trying to figure out how to configure rack awareness. I have written a script that reads a list of IPs or host names and return a list of rack IDs of the same number. This is my script running: $./mydns 192.168.1.1 192.168.2.1 /rack0 /rack1 I specified the path of this script to topology.script.file.name. I verified that this script was called by Hadoop and I could see the input (up to 21 IPs in my case). However, it seems the output of my script is not correct and Hadoop cannot use it to get the correct topology (only 1 rack is found by Hadoop no matter how I change the format of the output). Please advise if you know how to do this. Thanks Yunhong
Re: Combiner is optional though it is specified?
To my surprise, only one output value of mapper is not reaching combiner. and It is consistent when I repeated the experimentation. Same point directly reaches reducer without going thru the combiner. I am surprised how can this happen? novice user wrote: Regarding the conclusion, I am parsing the inputs in combiner and reducer differently. For example the output value of mapper is s:d where as the output value of combiner is s,d. So, in reducer, I am assuming the input as s,d and trying to parse it. There I got the exception because it got input as s:d. I am using hadoop-17. Icouldn't get exactly what you meant by no guarantee on the number of times a combiner is run. Can you please elaborate a bit on this? Thanks Arun C Murthy-2 wrote: On Jul 1, 2008, at 4:04 AM, novice user wrote: Hi all, I have a query regarding the functionality of combiner. Is it possible to ignore combiner code for some of the outputs of mapper and directly being sent to reducer though combiner is specified in job configuration? Because, I figured out that, when I am running on large amounts of data, some of the mapper output is directly reached reducer. I am wondering how can this be possible when I have specified combiner in the job configuration. Can any one please let me know if this thing happens? Can you elaborate on how you reached the conclusion that the output of some maps isn't going through the combiner? Also, what version of hadoop are you using? hadoop-0.18 onwards there aren't guarantees on the number of times a combiner is run... Arun -- View this message in context: http://www.nabble.com/Combiner-is- optional-though-it-is-specified--tp18213887p18213887.html Sent from the Hadoop core-user mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/Combiner-is-optional-though-it-is-specified--tp18213887p18254762.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Difference between joining and reducing
Hello all, After recent talk about joins, I have a (possibly) stupid question: What is the difference between the join operations in o.a.h.mapred.join and the standard merge step in a MapReduce job? I understand that doing a join in the Mapper would be much more efficient if you're lucky enough to have your input pre-sorted and -partitioned. But how is a join operation in the Reducer any different from the shuffle/sort/merge that the MapReduce framework already does? Be gentle. Thanks, -Stuart
Re: failed map tasks
I am actually more interested in _theoretically_ what could happen to a map tasks to fail or to take longer... Don't have a specific case. Thanks, Jerr Amar Kamat wrote: jerrro wrote: Hello, I was wondering - could someone tell me what are the reasons that I could get failure with certain map tasks on a node? Well, that depends on the kind of errors you are seeing. Could you plz post the logs/error messages? Amar Any idea that comes to mind would work (it does not happen because my program aborts). Also, what could cause with hadoop for a map task to take longer (although data is the same for all maps) to finish than the other tasks? What Hadoop version are you using? What is the average runtime of the maps? How much difference are you seeing? Are all the nodes same in terms of config? Thanks Jerr -- View this message in context: http://www.nabble.com/failed-map-tasks-tp18251144p18260183.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Help! How to overcome a RemoteException:
I have installed cygwin and hadoop-0.17.0 and have done 3 steps: 1)add JAVA_HOME in hadoop-env.sh 2)create hadoop-site.xml 3)execute commands: cd /cygdrive/c/hadoop-0.17.0 bin/start-all.sh bin/hadoop dfs -rmr input bin/hadoop dfs -put conf input NOT WORKING bin/hadoop dfs -ls bin/stop-all.sh the results are: cd /cygdrive/c/hadoop-0.17.0 bin/start-all.sh bin/hadoop dfs -rmr input bin/hadoop dfs -put conf input NOT WORKING bin/hadoop dfs -ls bin/stop-all.sh [EMAIL PROTECTED] ~ $ ./make-hadoop-site.sh [EMAIL PROTECTED] ~ $ cd /cygdrive/c/hadoop-0.17.0 [EMAIL PROTECTED] /cygdrive/c/hadoop-0.17.0 $ bin/start-all.sh starting namenode, logging to /cygdrive/c/hadoop-0.17.0/bin/../logs/hadoop-bstar chev-namenode-bis-laptop.out : no address associated with name localhost : no address associated with name localhost starting jobtracker, logging to /cygdrive/c/hadoop-0.17.0/bin/../logs/hadoop-bst archev-jobtracker-bis-laptop.out : no address associated with name localhost [EMAIL PROTECTED] /cygdrive/c/hadoop-0.17.0 $ bin/hadoop dfs -rmr input Deleted /user/bstarchev/input [EMAIL PROTECTED] /cygdrive/c/hadoop-0.17.0 $ bin/hadoop dfs -put conf inputNOT WORKING 08/06/21 19:19:55 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: jav a.io.IOException: File /user/bstarchev/input/commons-logging.properties could on ly be replicated to 0 nodes, instead of 1 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.ja va:1145) at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) at org.apache.hadoop.ipc.Client.call(Client.java:557) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI nvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat ionHandler.java:59) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock( DFSClient.java:2334) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream (DFSClient.java:2219) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient .java:1702) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSC lient.java:1842) 08/06/21 19:19:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/b starchev/input/commons-logging.properties retries left 4 08/06/21 19:19:56 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: jav a.io.IOException: File /user/bstarchev/input/commons-logging.properties could on ly be replicated to 0 nodes, instead of 1 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.ja va:1145) at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) at org.apache.hadoop.ipc.Client.call(Client.java:557) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI nvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat ionHandler.java:59) at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) at
RE: Difference between joining and reducing
Hi Stuart, Join is a higher level logical operation while map/reduce is a technique that could be used implement it. Specifically, in relational algebra, the join construct specifies how to form a single output row from 2 rows arising from two input streams. There are very many ways of implementing this logical operation and traditional database systems have a number of such implementations. Map/reduce being a system that essential allows you to cluster data by doing a distributed sort, is amenable to the sort based techinque for doing the join. A particular implementation of the reducer gets a combined stream of data from the two or more input streams such that they match on the key. It then proceeds to generate the cartesian product of the rows from the imput streams. In order to implement a join, you need to implement this join reducer yourself which is what org.apache.hadoop.mapred.join does. I hope that clears up the confusion. Cheers, Ashish -Original Message- From: [EMAIL PROTECTED] on behalf of Stuart Sierra Sent: Thu 7/3/2008 7:54 AM To: core-user@hadoop.apache.org Subject: Difference between joining and reducing Hello all, After recent talk about joins, I have a (possibly) stupid question: What is the difference between the join operations in o.a.h.mapred.join and the standard merge step in a MapReduce job? I understand that doing a join in the Mapper would be much more efficient if you're lucky enough to have your input pre-sorted and -partitioned. But how is a join operation in the Reducer any different from the shuffle/sort/merge that the MapReduce framework already does? Be gentle. Thanks, -Stuart
Re: Inconsistency in namenode's and datanode's namespaceID
Yes this is a known bug. http://issues.apache.org/jira/browse/HADOOP-1212 You should manually remove current directory from every data-node after reformatting the name-node and start the cluster again. I do not believe there is any other way. Thanks, --Konstantin Taeho Kang wrote: No, I don't think it's a bug. Your datanodes' data partition/directory was probably used in other HDFS setup and thus had other namespaceID. Or you could've used other partition/directory for your new HDFS setup by setting different values for dfs.data.dir on your datanode. But in this case, you can't access your old HDFS's data. On Thu, Jul 3, 2008 at 4:21 AM, Xuan Dzung Doan [EMAIL PROTECTED] wrote: I was following the quickstart guide to run pseudo-distributed operations with Hadoop 0.16.4. I got it to work successfully the first time. But I failed to repeat the steps (I tried to re-do everything from re-formating the HDFS). Then by looking at the log files of the daemons, I found out the datanode failed to start because its namespaceID didn't match with the namenode's. I after that found that the namespaceID is stored in the text file VERSION under dfs/data/current and dfs/name/current for the datanode and the namenode, respectively. The reformatting step does change namespaceID of the namenode, but not for the datanode, and that's the cause for the inconsistency. So after reformatting, if I manually update namespaceID for the datanode, things will work totally fine again. I guess there are probably others who had this same experience. Is it a bug in Hadoop 0.16.4? If so, has it been taken care of in later versions? Thanks, David.
Re: XEN guest OS
On Tuesday 01 July 2008 09:36:18 Ashok Varma wrote: Hi , I'm trying to install Fedora8 as a Guest OS in XEN on CentOS5.2 -64 bit. Always getting failed to Mount directory error. I configured NFS share, then also installation getting failed in middle.. Slightly offtopic on a hadoop mailing list? Andreas Any IDEA !!! signature.asc Description: This is a digitally signed message part.
Re: MapSide Join and left outer or right outer joins?
Forgive me if you already know this, but the correctness of the map- side join is very sensitive to partitioning; if your input in sorted but equal keys go to different partitions, your results may be incorrect. Is your input such that the default partitioning is sufficient? Have you verified the correctness of your results? -C On Jul 2, 2008, at 9:55 PM, Jason Venner wrote: For the data joins, I let the framework do it - which means one partition per split - so I have to chose my partition count carefully to fill the machines. I had an error in my initial outer join mapper, the join map code now runs about 40x faster than the old brute force read it all shuffle sort. Chris Douglas wrote: Hi Jason- It only seems like full outer or full inner joins are supported. I was hoping to just do a left outer join. Is this supported or planned? The full inner/outer joins are examples, really. You can define your own operations by extending o.a.h.mapred.join.JoinRecordReader or o.a.h.mapred.join.MultiFilterRecordReader and registering your new identifier with the parser by defining a property mapred.join.define.ident as your class. For a left outer join, JoinRecordReader is the correct base. InnerJoinRecordReader and OuterJoinRecordReader should make its use clear. On the flip side doing the Outer Join is about 8x faster than doing a map/reduce over our dataset. Cool! Out of curiosity, how are you managing your splits? -C
Re: Difference between joining and reducing
Ashish ably outlined the differences between a join and a merge, but might be confusing the o.a.h.mapred.join package and the contrib/ data_join framework. The former is used for map-side joins and has nothing to do with either the shuffle or the reduce; the latter effects joins in the reduce. The critical difference between the merge phase in map/reduce and a join is that merge outputs are grouped by a comparator and consumed in sorted order while, in contrast, joins involve n datasets and consumers will consider the cartesian product of selected keys (in both frameworks, equal keys). The practical differences between the two aforementioned join frameworks involve tradeoffs in efficiency and constraints on input data. -C On Jul 3, 2008, at 7:54 AM, Stuart Sierra wrote: Hello all, After recent talk about joins, I have a (possibly) stupid question: What is the difference between the join operations in o.a.h.mapred.join and the standard merge step in a MapReduce job? I understand that doing a join in the Mapper would be much more efficient if you're lucky enough to have your input pre-sorted and -partitioned. But how is a join operation in the Reducer any different from the shuffle/sort/merge that the MapReduce framework already does? Be gentle. Thanks, -Stuart
Re: MapSide Join and left outer or right outer joins?
We are using the default partitioner. I am just about to start verifying my result as it took quite a while to work my way through the in-obvious issues of hand writing MapFiles, thinks like the key and value class are extracted from the jobconf, output key/value. Question: I looked at the HashPartitioner (which we are using) and a key's partition is simply based on the key.hashCode() % conf.getNumReduces(). How will I get equal keys going to different partitions - clearly I have an understanding gap. Thanks! Chris Douglas wrote: Forgive me if you already know this, but the correctness of the map-side join is very sensitive to partitioning; if your input in sorted but equal keys go to different partitions, your results may be incorrect. Is your input such that the default partitioning is sufficient? Have you verified the correctness of your results? -C On Jul 2, 2008, at 9:55 PM, Jason Venner wrote: For the data joins, I let the framework do it - which means one partition per split - so I have to chose my partition count carefully to fill the machines. I had an error in my initial outer join mapper, the join map code now runs about 40x faster than the old brute force read it all shuffle sort. Chris Douglas wrote: Hi Jason- It only seems like full outer or full inner joins are supported. I was hoping to just do a left outer join. Is this supported or planned? The full inner/outer joins are examples, really. You can define your own operations by extending o.a.h.mapred.join.JoinRecordReader or o.a.h.mapred.join.MultiFilterRecordReader and registering your new identifier with the parser by defining a property mapred.join.define.ident as your class. For a left outer join, JoinRecordReader is the correct base. InnerJoinRecordReader and OuterJoinRecordReader should make its use clear. On the flip side doing the Outer Join is about 8x faster than doing a map/reduce over our dataset. Cool! Out of curiosity, how are you managing your splits? -C -- Jason Venner Attributor - Program the Web http://www.attributor.com/ Attributor is hiring Hadoop Wranglers and coding wizards, contact if interested
Re: Getting stats of running job from within job
Nathan Marz wrote: Is there a way to get stats of the currently running job programatically? This should probably be an FAQ. In your Mapper or Reducer's configure implementation, you can get a handle on the running job with: RunningJob running = new JobClient(job).getJob(job.get(mapred.job.id)); Doug
RE: topology.script.file.name
This is my script, which is actually a C++ program:
#include iostream
#include string
using namespace std;
int main(int argc, char** argv)
{
for (int i = 1; i argc; i ++ )
{
string dn = argv[i];
if (dn.substr(0, 5) == rack1)
cout /rack1;
else if (dn.substr(0, 5) == rack2)
cout /rack2;
else if (dn.substr(0, 3) == 192)
cout /rack1;
else if (dn.substr(0, 2) == 10)
cout /rack2;
else
cout /rack0;
cout ;
}
return 1;
}
So I compiled the program as mydns. It can accept many IPs and print
/rack0, /rack1, or /rack2 in a row.
e.g.,
./mydns 192.168.0.1 10.0.0.1
/rack1 rack2
(I tried other possible output, like each rack ID in one row, which
didn't help)
I configured hadoop-site.xml and add this
property
nametopology.script.file.name/name
value/home/my/hadoop-0.17.0/mydns/value
/property
The program is located at /home/my/hadoop-0.17.0.
My understanding is that mydns should be called by
ScriptBasedMapping.java.
I added some output to file in the mydns program and I can verify that it
is actually being called, with an input parameter something like
192.168.0.1 192.168.0.10 10.0.0.5.
However, when I ran ./bin/hadoop fsck, it still tells me that there is
only one rack in the system, and MapReduce program will immediately fail
because some topology initialization error (I could find the exact text
any more).
Thanks
Yunhong
On Thu, 3 Jul 2008, Devaraj Das wrote:
This is strange. If you don't mind, pls send the script to me.
-Original Message-
From: Yunhong Gu1 [mailto:[EMAIL PROTECTED]
Sent: Thursday, July 03, 2008 9:49 AM
To: core-user@hadoop.apache.org
Subject: topology.script.file.name
Hello,
I have been trying to figure out how to configure rack
awareness. I have written a script that reads a list of IPs
or host names and return a list of rack IDs of the same number.
This is my script running:
$./mydns 192.168.1.1 192.168.2.1
/rack0 /rack1
I specified the path of this script to
topology.script.file.name. I verified that this script was
called by Hadoop and I could see the input (up to 21 IPs in my case).
However, it seems the output of my script is not correct and
Hadoop cannot use it to get the correct topology (only 1 rack
is found by Hadoop no matter how I change the format of the output).
Please advise if you know how to do this.
Thanks
Yunhong
Help: how to check the active datanodes?
Hi guys: I am running hadoop on a 8 nodes cluster. I uses start-all.sh to boot hadoop and it shows that all 8 data nodes are started. However, when I use bin/hadoop dfsadmin -report to check the status of the data nodes and it shows only one data node (the one with the same host as name node) is active. How could we know if all the data nodes are active precisely? Does anyone has deal with this before? Thanks. Richard
ERROR dfs.NameNode - java.io.EOFException
Hi, Using Hadoop 0.16.2, I am seeing seeing the following in the NN log: 2008-07-03 19:46:26,715 ERROR dfs.NameNode - java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106) at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90) at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:433) at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:756) at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:639) at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:222) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:254) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:235) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:176) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:162) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:846) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:855) The exception doesn't include the name and location of the file whose reading is failing and causing EOFException :( But it looks like it's the fsedit log (the edits file, I think). There is no secondary NN in the cluster. Is there any way I can revive this NN? Any way to fix the corrupt edits file? Thanks, Otis
Hudson Patch Verifier's Output
Hi, I submitted a patch using JIRA and the Hudson system told that -1 contrib tests. The patch failed contrib unit tests. Seeing the console output, I noticed that it says build successful for contrib tests. So I am confused that what failed contrib test are referred to in Hudson output? This link https://issues.apache.org/jira/browse/HADOOP-3646 has the comments produced by Hudson patch verifier. Thanks, Abdul Qadeer
Help regarding LoginException - CreateProcess: whoami error=2
Hello, I am new to Hadoop and am trying to run HadoopDfsReadWriteExamplehttp://wiki.apache.org/hadoop/HadoopDfsReadWriteExample?action=fullsearchvalue=linkto%3A%22HadoopDfsReadWriteExample%22context=180 from eclipse on Windows XP. I have added following files in the build path for the project: commons-logging-1.0.4.jar hadoop-0.16.4-core.jar log4j-1.2.11.jar I am receiving following exception while trying to access the file system corresponding to my configuration files . Note, that HDFS is up and I can put files on HDFS from cygwin and browse HDFS using the web interface. Can someone pls help me resolve this issue? I understand that it is not able to execute whoami, but I do not know how I can execute whoami from eclipse successfully. Also, pls note that there is no space in my user name. Thanks in advance for your time, Rutuja == 08/07/03 15:40:48 WARN fs.FileSystem: uri=hdfs://localhost:7072 javax.security.auth.login.LoginException: Login failed: CreateProcess: whoami error=2 at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257) at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67) at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:1255) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1191) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:150) at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:96) at com.flurry.HDFSManager.HDFSFileReadWrite.main(HDFSFileReadWrite.java:68) Exception in thread main java.io.IOException at org.apache.hadoop.dfs.DFSClient.init(DFSClient.java:148) at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:65) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1182) at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:55) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1193) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:150) at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:96) at com.flurry.HDFSManager.HDFSFileReadWrite.main(HDFSFileReadWrite.java:68) Caused by: javax.security.auth.login.LoginException: Login failed: CreateProcess: whoami error=2 at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275) at org.apache.hadoop.dfs.DFSClient.init(DFSClient.java:146) ... 8 more
Re: Hudson Patch Verifier's Output
A bug was introduced by HADOOP-3480. HADOOP-3653 will fix it. Nige On Jul 3, 2008, at 5:24 PM, Abdul Qadeer wrote: Hi, I submitted a patch using JIRA and the Hudson system told that -1 contrib tests. The patch failed contrib unit tests. Seeing the console output, I noticed that it says build successful for contrib tests. So I am confused that what failed contrib test are referred to in Hudson output? This link https://issues.apache.org/jira/browse/HADOOP-3646 has the comments produced by Hudson patch verifier. Thanks, Abdul Qadeer
Re: Volunteer recruitment for RDF store project on Hadoop
Thanks for all interest. BTW, I can't handle too many people via private email , Please join this group. http://groups.google.com/group/hrdfstore Thanks, Edward On Wed, Jul 2, 2008 at 3:06 PM, Edward J. Yoon [EMAIL PROTECTED] wrote: Hello all, The HRdfStore team looking for a couple more volunteers. We'll develop a Hadoop subsystem for RDF, called HrdfStore, which uses Hadoop + Hbase + MapReduce to store RDF data and execute queries (e.g., SPARQL) on them. If you interested in here contact me at [EMAIL PROTECTED] Thanks. -- Best regards, Edward J. Yoon, http://blog.udanax.org -- Best regards, Edward J. Yoon, http://blog.udanax.org
Re: Help: how to check the active datanodes?
Hi, zhang: Once you start hadoop with shell start-all.sh, a hadoop status pape can be accessed through http://namenode-ip:port/dfshealth. Port is specified by namedfs.http.address/name in your hadoop-default.xml. If the datanodes status is not as expected, you need to check log files. They show the details of failure. On Fri, Jul 4, 2008 at 4:17 AM, Richard Zhang [EMAIL PROTECTED] wrote: Hi guys: I am running hadoop on a 8 nodes cluster. I uses start-all.sh to boot hadoop and it shows that all 8 data nodes are started. However, when I use bin/hadoop dfsadmin -report to check the status of the data nodes and it shows only one data node (the one with the same host as name node) is active. How could we know if all the data nodes are active precisely? Does anyone has deal with this before? Thanks. Richard -- [EMAIL PROTECTED] Institute of Computing Technology, Chinese Academy of Sciences, Beijing.
nested for loops
I'm a newbie, so feel free to rftm is this is old hat: what's the best way to do a nested for loop in hadoop? Specifically, lets say I've got a list of elements, and I want to do an all against all comparison. The standard nested for loop would be: for i in 1..10: for j in i..10: doSomething(myList[i],myList[j]) Any good ways to do this in hadoop? I assume I could do the full all-against-all with a nested map: map1 key elements, value elements map2key elements, value elements Is there any way to not do the duplicate calculations. This is a pretty common code pattern, so I figure someone has thought this through before. (Also, is it kosher to have a map function that calls another map function or will that mess up the scheduler?)
