Steve Loughran created HADOOP-15240:
---------------------------------------
Summary: ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths
failure, FileAlreadyExistsException
Key: HADOOP-15240
URL: https://issues.apache.org/jira/browse/HADOOP-15240
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.1.0, 3.2.0
Reporter: Steve Loughran
Transient {{ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths}} failure;
{{FileAlreadyExistsException}} during the setup phase of the test -writing out
the work to
{{org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:
file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/build/test:fork-0002/job_200707120153_0002/_temporary/1/_temporary/attempt_200707120153_0002_m_000000_0/part-m-00000}}
There's something wrong here as that path is a FileOutputCommitter workingDir
path, not a partitioned committer.
Hypotheses: some filesystem caching in the test run is stopping the right
committer from being picked up.
It's possible that this has been happening for a while, but only some cleanup
failure from a previous run has caught it.
Proposed
* add checks in the AbstractIT test to verify the right committer is picked up
in the task context by probing for the working dir: file:// for staging,
*/__magic/* for magic.
* and verify that success file is there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
