Re: s3n exceptions
Are you using 0.18? I know that the copy HDFS to s3n isn't supported there yet. I think there's a fix in 0.19. / Per On Mon, Nov 24, 2008 at 2:11 AM, Alexander Aristov [EMAIL PROTECTED] wrote: Hi all I am testing s3n file system facilities and try to copy from hdfs to S3 in original format And I get next errors 08/11/24 05:04:49 INFO mapred.JobClient: Running job: job_200811240437_0004 08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0% 08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0% 08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0% 08/11/24 05:05:03 INFO mapred.JobClient: Task Id : attempt_200811240437_0004_m_00_0, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 7 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) 08/11/24 05:05:15 INFO mapred.JobClient: Task Id : attempt_200811240437_0004_m_00_1, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 7 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) -- Best Regards Alexander Aristov
Re: s3n exceptions
I'm thinking of this bug: http://issues.apache.org/jira/browse/HADOOP-3361 It was the cause of problems I hit trying to copy from HDFS to S3N with 0.18.1. / Per On Wed, Nov 26, 2008 at 12:57 PM, Alexander Aristov [EMAIL PROTECTED] wrote: yes, I am trying 0.18.2 But according to hadoop wiki it is supported http://wiki.apache.org/hadoop/AmazonS3 and http://issues.apache.org/jira/browse/HADOOP-930 Alexander 2008/11/26 Per Jacobsson [EMAIL PROTECTED] Are you using 0.18? I know that the copy HDFS to s3n isn't supported there yet. I think there's a fix in 0.19. / Per On Mon, Nov 24, 2008 at 2:11 AM, Alexander Aristov [EMAIL PROTECTED] wrote: Hi all I am testing s3n file system facilities and try to copy from hdfs to S3 in original format And I get next errors 08/11/24 05:04:49 INFO mapred.JobClient: Running job: job_200811240437_0004 08/11/24 05:04:50 INFO mapred.JobClient: map 0% reduce 0% 08/11/24 05:05:00 INFO mapred.JobClient: map 44% reduce 0% 08/11/24 05:05:03 INFO mapred.JobClient: map 0% reduce 0% 08/11/24 05:05:03 INFO mapred.JobClient: Task Id : attempt_200811240437_0004_m_00_0, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 7 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) 08/11/24 05:05:15 INFO mapred.JobClient: Task Id : attempt_200811240437_0004_m_00_1, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 7 at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:542) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) -- Best Regards Alexander Aristov -- Best Regards Alexander Aristov
Re: error with inout for the Reduce method
Change the name of the reduce method to be all lower case -- public void
reduce(...
Right now the compiler is complaining that you haven't overridden the
correct abstract method in the base class.
/ Per
On Sat, Nov 8, 2008 at 10:44 PM, pols cut [EMAIL PROTECTED] wrote:
I am trying to get a simple map reduce function going but getting the
following error. could anyone please help/guide me
filename.java is not abstract and does not override abstract method
reduceorg.apache.io.text,java.util.iteratororg.apache.hadoop.io.text,org.apache.hadoop.mapred.OutputCollectororg.apache.hadoop.io.Text.org.apache.hadoop.mapred.Reporter
in org.apache.hadoop.mapred.Reducer
public static class Reduce extends MapReduceBase implements
ReducerText,Text,Text,Text
My Map method is like this:
public static class map extends MapReduceBase implements
MapperLongWritable, Text, Text, Text
{
public void map(LongWritable key, Text value, OutputCollectorText, Text
output, Reporter reporter) throws IOException
{
output.collect(word_key,new Text(str1)); //output.collect(Text,Text)
}
}
public static class Reduce extends MapReduceBase implements ReducerText,
Text, Text, Text
{
public void Reduce(Text key, IteratorText values,
OutputCollectorText, Text output, Reporter reporter) throws IOException
{
output.collect(new Text(Test), new Text(Me));
}
}
configuration is like :
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Text.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
conf.setMapperClass(map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
Add more friends to your messenger and enjoy! Go to
http://messenger.yahoo.com/invite/
Re: Auto-shutdown for EC2 clusters
We're doing the same thing, but doing the scheduling just with shell scripts running on a machine outside of the Hadoop cluster. It works but we're getting into a bit of scripting hell as things get more complex. We're using distcp to first copy the files the jobs need from S3 to HDFS and it works nicely. When going the other direction we have to pull the data down from HDFS to one of the EC2 machines and then push back it up to S3. If I understand things right the support for that will be better in 0.19. / Per On Thu, Oct 23, 2008 at 8:52 AM, Chris K Wensel [EMAIL PROTECTED] wrote: Hey Stuart I did that for a client using Cascading events and SQS. When jobs completed, they dropped a message on SQS where a listener picked up new jobs and ran with them, or decided to kill off the cluster. The currently shipping EC2 scripts are suitable for having multiple simultaneous clusters for this purpose. Cascading has always and now Hadoop supports (thanks Tom) raw file access on S3, so this is quite natural. This is the best approach as data is pulled directly into the Mapper, instead of onto HDFS first, then read into the Mapper from HDFS. YMMV chris On Oct 23, 2008, at 7:47 AM, Stuart Sierra wrote: Hi folks, Anybody tried scripting Hadoop on EC2 to... 1. Launch a cluster 2. Pull data from S3 3. Run a job 4. Copy results to S3 5. Terminate the cluster ... without any user interaction? -Stuart -- Chris K Wensel [EMAIL PROTECTED] http://chris.wensel.net/ http://www.cascading.org/
Expected disk space use for files in mapred/local?
Hi all, We've been running a pretty big job on 20 extra-large high-CPU EC2 servers (Hadoop version 0.18, Java 1.6, the standard AMIs), and started getting the dreaded Could not find any valid local directory error during the final reduce phase. I've confirmed that some of the boxes are running out of space, but the disk usage seems to be very uneven across the servers. The datanodes report 50% of available space used on all servers, which matches what I'm seeing in the /mnt/hadoop/dfs/data/folder (an even ~200 Gb / server). But the space used by files in /mnt/hadoop/mapred/local differs a lot from server to server (going from 70 Gb to 190 Gb). Is there any way to predict how much space will be used by the temporary data stored outside of HDFS? We're only running a total of 20 reducers which I suspect is very low, since there are a few thousand map tasks. Could that be the cause or is there anything else we're doing that's obviously wrong? Besides this, we're also getting this error: java.lang.OutOfMemoryError: GC overhead limit exceeded Thanks for any help, / Per
Re: Merging of the local FS files threw an exception
Quick FYI: I've run the same job twice more without seeing the error. / Per On Wed, Oct 1, 2008 at 11:07 AM, Per Jacobsson [EMAIL PROTECTED] wrote: Hi everyone, (apologies if this gets posted on the list twice for some reason, my first attempt was denied as suspected spam) I ran a job last night with Hadoop 0.18.0 on EC2, using the standard small AMI. The job was producing gzipped output, otherwise I haven't changed the configuration. The final reduce steps failed with this error that I haven't seem before: 2008-10-01 05:02:39,810 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200809301822_0005_r_01_0 Merging of the local FS files threw an exception: java.io.IOException: java.io.IOException: Rec# 289050: Negative value-length: -96 at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:331) at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:134) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:225) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2021) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2025) 2008-10-01 05:02:44,131 WARN org.apache.hadoop.mapred.TaskTracker: Error running child java.io.IOException: attempt_200809301822_0005_r_01_0The reduce copier failed at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209) When I try to download the data from HDFS I get a Found checksum error warning message. Any ideas what could be the cause? Would upgrading to 0.18.1 solve it? Thanks, / Per
Re: Merging of the local FS files threw an exception
I've collected the syslogs from the failed reduce jobs. What's the best way to get them to you? Let me know if you need anything else, I'll have to shut down these instances some time later today. Overall I've run this same job before with no problems. The only change is the added gzip of the output. Don't know if it's worth anything, but the four failures all happened on different machines. I'll be running this job plenty of times so if the problem keeps happening it will be obvious. / Per On Wed, Oct 1, 2008 at 11:23 AM, Arun C Murthy [EMAIL PROTECTED] wrote: Do you still have the task logs for the reduce? I suspect are running into http://issues.apache.org/jira/browse/HADOOP-3647 which we never could reproduce reliably to pin it down or fix. However, in light of http://issues.apache.org/jira/browse/HADOOP-4277 we suspect this could be caused by a bug in the LocalFileSystem which could hide data-corruption on your local disk leading to errors on these nature. Could you try running your job with that patch once the release 0.18.2 is available? Any information you provide could greatly aid to confirm our above hypothesis, so it's much appreciated! Arun
Re: Merging of the local FS files threw an exception
Attached to the ticket. Hope this helps. / Per On Wed, Oct 1, 2008 at 1:33 PM, Arun C Murthy [EMAIL PROTECTED] wrote: On Oct 1, 2008, at 12:04 PM, Per Jacobsson wrote: I've collected the syslogs from the failed reduce jobs. What's the best way to get them to you? Let me know if you need anything else, I'll have to shut down these instances some time later today. Could you please attach them to the jira: http://issues.apache.org/jira/browse/HADOOP-3647? Thanks! Arun
Re: Do all Mapper outputs with same key go to same Reducer?
If that's true, then can I set the number of Reducers very high (even equal to the number of maps) to make Job C go faster? This page has some good info on finding the right number of reducers: http://wiki.apache.org/hadoop/HowManyMapsAndReduces / Per On Fri, Sep 19, 2008 at 9:42 AM, Miles Osborne [EMAIL PROTECTED] wrote: So here's my question -- does Hadoop guarantee that all records with the same key will end up in the same Reducer task? If that's true, yes --think of the record as being sent to the task by hashing over the key Miles 2008/9/19 Stuart Sierra [EMAIL PROTECTED]: Hi all, The short version of my question is in the subject. Here's the long version: I have two map/reduce jobs that output records using a common key: Job A: K1 = A1,1 K1 = A1,2 K2 = A2,1 K2 = A2,2 Job B: K1 = B1 K2 = B2 K3 = B3 And a third job that merges records with the same key, using IdentityMapper and a custom Reducer: Job C: K1 = A1,1; A2,2; B1 K2 = A2,1; A2,2; B2 K3 = B3 The trouble is, the A's and B's are large (20-30 KB each) and I have a few million of them. If Job C has only one Reducer task, it takes forever to copy and sort all the records. Thanks for any enlightenment you can provide here, -Stuart
