[
https://issues.apache.org/jira/browse/HDDS-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192839#comment-17192839
]
Marton Elek commented on HDDS-4209:
-----------------------------------
I checked it with aws s3 api, and after the first step I can see a good entry:
{code}
aws s3api list-objects --bucket ozonetest --prefix=o11
{
"Contents": [
{
"Key": "o11/o12/",
"LastModified": "2020-09-09T12:34:36.000Z",
"ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
"Size": 0,
"StorageClass": "STANDARD",
"Owner": {
"DisplayName": "e1428",
"ID":
"b8c021b4343e316b28b545df160c6720479a998001ebf7019328b64417fe152d"
}
}
]
}
{code}
As the `/` postfix is added to the path the intermediate directory creation
logic can understand it's a directory and can reuse the object. IMHO.
> S3A Filesystem does not work with Ozone S3
> ------------------------------------------
>
> Key: HDDS-4209
> URL: https://issues.apache.org/jira/browse/HDDS-4209
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Blocker
>
> When *ozone.om.enable.filesystem.paths* is enabled
>
> hdfs dfs -mkdir -p s3a://b12345/d11/d12 -> Success
> hdfs dfs -put /tmp/file1 s3a://b12345/d11/d12/file1 -> fails with below error
>
> {code:java}
> 2020-09-04 03:53:51,377 ERROR
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest: Key creation
> failed. Volume:s3v, Bucket:b1234, Keyd11/d12/file1._COPYING_. Exception:{}
> NOT_A_FILE org.apache.hadoop.ozone.om.exceptions.OMException: Can not create
> file: cp/k1._COPYING_ as there is already file in the given path
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:256)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:428)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:246)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
> *Reason for this*
> S3A filesystem when create directory creates an empty file
> *Now entries in Ozone KeyTable after create directory*
> d11/
> d11/d12
> Because of this in OMFileRequest.VerifyInFilesPath fails with
> FILE_EXISTS_IN_GIVEN_PATH because d11/d12 is considered as file not a
> directory. (As in ozone currently, directories end with trailing "/")
> So, when d11/d12/file is created, we check parent exists, now d11/d12 is
> considered as file and fails with NOT_A_FILE
> When disabled it works fine, as when disabled during key create we do not
> check any filesystem semantics and also does not create intermediate
> directories.
> {code:java}
> [root@bvoz-1 ~]# hdfs dfs -mkdir -p s3a://b12345/d11/d12
> [root@bvoz-1 ~]# hdfs dfs -put /etc/hadoop/conf/ozone-site.xml
> s3a://b12345/d11/d12/k1
> [root@bvoz-1 ~]# hdfs dfs -ls s3a://b12345/d11/d12
> Found 1 items
> -rw-rw-rw- 1 systest systest 2373 2020-09-04 04:45
> s3a://b12345/d11/d12/k1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
