issue creating buffer directory when using native S3FileSystem
--------------------------------------------------------------
Key: HADOOP-5802
URL: https://issues.apache.org/jira/browse/HADOOP-5802
Project: Hadoop Core
Issue Type: Bug
Environment: ec2, Cloudera AMI, m-xlarge, 20 nodes.
Reporter: Arun Jacob
Note the following settings in hadoop-site.xml, which means I've configured 40
reduce tasks to run across 20 nodes. I checked Jira, and found HADOOP-4377,
which refers to this as a race condition that has been fixed, the bug was
closed. I'm re-opening because this may be configuration specific, e.g. I am
running multiple reduce tasks on the same node (no mention was made of that in
the original bug)
<property>
<name>mapred.reduce.tasks</name>
<value>40</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>
<final>true</final>
</property>
One of the tasks fails trying to create the S3 buffer directory:
attempt_200905111058_0001_r_000019_0: 1589020 [main] WARN
org.apache.hadoop.mapred.TaskTracker - Error running child
attempt_200905111058_0001_r_000019_0: java.io.IOException: Cannot create S3
buffer directory: /mnt/tmp/hadoop-hadoop/s3
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.newBackupFile(NativeS3FileSystem.java:152)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:136)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.create(NativeS3FileSystem.java:278)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:503)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:403)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:117)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:293)
attempt_200905111058_0001_r_000019_0: at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
