adoroszlai opened a new pull request #2462:
URL: https://github.com/apache/hadoop/pull/2462
## What changes were proposed in this pull request?
Fix ITestS3AContractRename failure:
```
[ERROR]
testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 2.805 s <<< FAILURE!
java.lang.AssertionError: expected rename(s3a://rename/test/source-256.txt,
s3a://rename/test/dest-512.txt) to be rejected with exception, but got false
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:131)
```
by declaring `rename-returns-false-if-dest-exists=true` in the contract.
https://issues.apache.org/jira/browse/HADOOP-17376
## How was this patch tested?
Ran `ITestS3AContractRename` against Ozone S3 Gateway.
1. Download, build and run Ozone:
```
git clone https://github.com/apache/ozone
cd ozone
git checkout a28cda57e
mvn -DskipTests -Dskip.installnpx -Dskip.npx -DskipShade clean package
cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/ozone
docker-compose up -d --scale datanode=3
```
2. Wait about a minute for Ozone to start.
3. Create bucket for test:
```
aws s3api create-bucket --bucket rename --endpoint http://localhost:9878
```
4. Configure test to run against Ozone:
```
<property>
<name>fs.s3a.endpoint</name>
<value>http://127.0.0.1:9878/</value>
</property>
<property>
<name>test.fs.s3a.name</name>
<value>s3a://rename/</value>
</property>
```
5. Run test in Hadoop source:
```
cd ~/src/hadoop
mvn -pl :hadoop-aws -Dtest='ITestS3AContractRename' test
```
Now `ITestS3AContractRename#testRenameFileOverExistingFile` proceeds beyond
previous failure in line 131, and fails due to Ozone-specific incompatibility
in line 146:
```
[ERROR]
testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 3.468 s <<< ERROR!
org.apache.hadoop.fs.s3a.NoVersionAttributeException:
`s3a://rename/test/dest-512.txt': Change detection policy requires ETag
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:275)
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:261)
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:195)
at
org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:208)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:347)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:343)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.contract.ContractTestUtils.readDataset(ContractTestUtils.java:216)
at
org.apache.hadoop.fs.contract.ContractTestUtils.verifyFileContents(ContractTestUtils.java:240)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:146)
```
I expect this to work on real S3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
