[
https://issues.apache.org/jira/browse/HADOOP-17376?focusedWorklogId=510954&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-510954
]
ASF GitHub Bot logged work on HADOOP-17376:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Nov/20 18:11
Start Date: 12/Nov/20 18:11
Worklog Time Spent: 10m
Work Description: adoroszlai opened a new pull request #2462:
URL: https://github.com/apache/hadoop/pull/2462
## What changes were proposed in this pull request?
Fix ITestS3AContractRename failure:
```
[ERROR]
testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 2.805 s <<< FAILURE!
java.lang.AssertionError: expected rename(s3a://rename/test/source-256.txt,
s3a://rename/test/dest-512.txt) to be rejected with exception, but got false
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:131)
```
by declaring `rename-returns-false-if-dest-exists=true` in the contract.
https://issues.apache.org/jira/browse/HADOOP-17376
## How was this patch tested?
Ran `ITestS3AContractRename` against Ozone S3 Gateway.
1. Download, build and run Ozone:
```
git clone https://github.com/apache/ozone
cd ozone
git checkout a28cda57e
mvn -DskipTests -Dskip.installnpx -Dskip.npx -DskipShade clean package
cd hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/compose/ozone
docker-compose up -d --scale datanode=3
```
2. Wait about a minute for Ozone to start.
3. Create bucket for test:
```
aws s3api create-bucket --bucket rename --endpoint http://localhost:9878
```
4. Configure test to run against Ozone:
```
<property>
<name>fs.s3a.endpoint</name>
<value>http://127.0.0.1:9878/</value>
</property>
<property>
<name>test.fs.s3a.name</name>
<value>s3a://rename/</value>
</property>
```
5. Run test in Hadoop source:
```
cd ~/src/hadoop
mvn -pl :hadoop-aws -Dtest='ITestS3AContractRename' test
```
Now `ITestS3AContractRename#testRenameFileOverExistingFile` proceeds beyond
previous failure in line 131, and fails due to Ozone-specific incompatibility
in line 146:
```
[ERROR]
testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)
Time elapsed: 3.468 s <<< ERROR!
org.apache.hadoop.fs.s3a.NoVersionAttributeException:
`s3a://rename/test/dest-512.txt': Change detection policy requires ETag
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:275)
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:261)
at
org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:195)
at
org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:208)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:347)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:343)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.contract.ContractTestUtils.readDataset(ContractTestUtils.java:216)
at
org.apache.hadoop.fs.contract.ContractTestUtils.verifyFileContents(ContractTestUtils.java:240)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:146)
```
I expect this to work on real S3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 510954)
Remaining Estimate: 0h
Time Spent: 10m
> ITestS3AContractRename failing against stricter tests
> -----------------------------------------------------
>
> Key: HADOOP-17376
> URL: https://issues.apache.org/jira/browse/HADOOP-17376
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Attila Doroszlai
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HADOOP-17365 tightened the contract test for rename over file; S3A is failing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
