[
https://issues.apache.org/jira/browse/HADOOP-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16177.
-------------------------------------
Resolution: Invalid
some logging shows up the problem: I'd switched my (common) core-site to use
the staging committer by default
{code}
<property>
<name>fs.s3a.committer.name</name>
<value>directory</value>
<description>
Committer to create for output to S3A, one of:
"file", "directory", "partitioned", "magic".
</description>
</property>
{code}
And that was working -but generating unique filenames, as it does by default.
Simple fix: turn off unique filenames.
{code}
<property>
<name>fs.s3a.committer.staging.unique-filenames</name>
<value>false</value>
</property
{code}
I was worried about some serious problem which s3guard was either not picking
up or somehow creating, but no, just configuration changes.
> regression: ITestS3AMiniYarnCluster failing on branch-3.2
> ---------------------------------------------------------
>
> Key: HADOOP-16177
> URL: https://issues.apache.org/jira/browse/HADOOP-16177
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Priority: Major
>
> The test ITestS3AMiniYarnCluster is failing -the output of the trivial MR job
> isn't being found.
> {code}
> tS3AMiniYarnCluster
> [ERROR]
> testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster)
> Time elapsed: 8.309 s <<< ERROR!
> java.io.FileNotFoundException: No such file or directory:
> s3a://hwdev-steve-ireland-new/test/MiniClusterWordCount/output/part-r-00000
> at
> org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.readStringFromFile(ITestS3AMiniYarnCluster.java:144)
> at
> org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.testWithMiniCluster(ITestS3AMiniYarnCluster.java:103)
> {code}
> In trunk, HADOOP-14556 moved this test to the new committers, and the problem
> doesn't surface.
> Thought this was related to S3Guard not on, or the fact that a versioned S3
> store was in use (S3 london), but it also arises (at least today) on S3
> ireland with s3guard and no versioned store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
