Re: RandomTextWriter
fyi, Things seem to be playing nicer on Hadoop 0.17.1. But I'm also now running on c1.medium EC2 instances with the recommended XEN kernel. So that could be a factor as well. ckw On Jul 7, 2008, at 10:30 AM, Chris K Wensel wrote: In local mode, only one mapper succeeds, the remaining never start. And when on a cluster, a handful of mappers die with some exception about not finding the output directory (sorry, don't have the exception handy). I'm upgrading to 0.17.1 to see if they persist. ckw On Jul 7, 2008, at 10:08 AM, Arun C Murthy wrote: On Jul 7, 2008, at 9:46 AM, Chris K Wensel wrote: Hey all Has anyone had success with RandomTextWriter? I'm finding it fairly unstable on 0.16.x, haven't tried 0.17 yet though. What problems are you seeing? It seems to work fine for me... Arun -- Chris K Wensel [EMAIL PROTECTED] http://chris.wensel.net/ http://www.cascading.org/ -- Chris K Wensel [EMAIL PROTECTED] http://chris.wensel.net/ http://www.cascading.org/
Re: RandomTextWriter
In local mode, only one mapper succeeds, the remaining never start. And when on a cluster, a handful of mappers die with some exception about not finding the output directory (sorry, don't have the exception handy). I'm upgrading to 0.17.1 to see if they persist. ckw On Jul 7, 2008, at 10:08 AM, Arun C Murthy wrote: On Jul 7, 2008, at 9:46 AM, Chris K Wensel wrote: Hey all Has anyone had success with RandomTextWriter? I'm finding it fairly unstable on 0.16.x, haven't tried 0.17 yet though. What problems are you seeing? It seems to work fine for me... Arun -- Chris K Wensel [EMAIL PROTECTED] http://chris.wensel.net/ http://www.cascading.org/
Re: RandomTextWriter
On Jul 7, 2008, at 9:46 AM, Chris K Wensel wrote: Hey all Has anyone had success with RandomTextWriter? I'm finding it fairly unstable on 0.16.x, haven't tried 0.17 yet though. What problems are you seeing? It seems to work fine for me... Arun
RandomTextWriter
Hey all Has anyone had success with RandomTextWriter? I'm finding it fairly unstable on 0.16.x, haven't tried 0.17 yet though. chris -- Chris K Wensel [EMAIL PROTECTED] http://chris.wensel.net/ http://www.cascading.org/
randomtextwriter can not write 100GB text file on a 500GB cluster
Hi folks: I am trying to write a 100GB text file on a cluster with 500GB free storage space. For smaller scale writing such as 100MB, 1G, it works fine. But it shows the DFS client can not complete writing errors when I ran with 100GB writing. Does anyone has any ideas on these types of errors or met it before? Below is the command I used: bin/hadoop jar hadoop-0.17.0-examples.jar randomtextwriter -D test.randomtextwrite.min_words_key=5 -D test.randomtextwrite.max_words_key=10 -D test.randomtextwrite.min_words_value=20 test.randomtextwrite.max_words_value=100 -D test.randomtextwrite.total_bytes=107374182000 rand-text. And the below are the errors: 08/06/29 14:18:18 INFO mapred.JobClient: map 1% reduce 0% 08/06/29 14:18:20 INFO mapred.JobClient: Task Id : task_200806291409_0001_m_12_0, Status : FAILED org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not complete write to file /user/hadoop/rand-text/_temporary/_task_200806291409_0001_m_12_0/part-00012 by DFSClient_task_200806291409_0001_m_12_0 at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) at org.apache.hadoop.ipc.Client.call(Client.java:557) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655) at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79) at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:969) at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79) at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:257) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:224) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
