[
https://issues.apache.org/jira/browse/HDDS-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654247#comment-17654247
]
Ethan Rose commented on HDDS-7594:
----------------------------------
Looks like this is going to happen when connecting HDFS services to Ozone via
s3a then s3 gateway, since anything using s3 natively would be unlikely to do
this. This is not ideal since you could run those services directly with OFS,
but it is a use case we support.
I think the config could be added under the `ozone.s3g` prefix . When enabled,
it would check the bucket layout type of the `OzoneBucket` in
`ObjectEndpoint#put`. If it is FSO, the request is to write a 0 byte file, and
the config is enabled, s3g would send a directory create request instead of a
file write request to Ozone.
Since this would only apply to s3 gateway and FSO buckets, we may be able to
get away without a config for simplicity. If we do decide to add a config for
this I think it should be on (create directory on empty file put) by default.
> [FSO] Folders created through S3G are created on file system as "files".
> ------------------------------------------------------------------------
>
> Key: HDDS-7594
> URL: https://issues.apache.org/jira/browse/HDDS-7594
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Neil Joshi
> Priority: Major
>
> With FSO bucket layout buckets, folders created with the s3-gateway create 0
> byte files on file system. Unlike Object Store bucket layout buckets that
> create folders correctly when created through the s3-gateway, folder creation
> on FSO buckets with s3 gateway results in 0 byte files. Expected is the same
> behaviour. This has caused problems with interoperability for applications
> accessing ozone through both ofs and the s3 gateway, such as Trino.
>
> Problem can be produced issuing aws-cli folder creation to the s3 gateway on
> an ozone cluster layout {*}FILE_SYSTEM_OPTIMIZED{*}:
>
> {code:java}
> bash-4.2$ ozone sh bucket create --layout=FILE_SYSTEM_OPTIMIZED /s3v/bucketfso
> ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object
> --bucket bucketfso --key test1
> ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object
> --bucket bucketfso --key test1/README.md --body ./README.md
> An error occurred (InvalidRequest) when calling the PutObject operation: An
> error occurred (InvalidRequest) when calling the PutObject/MPU PartUpload
> operation: ozone.om.enable.filesystem.paths is enabled Keys are considered as
> Unix Paths. Path has Violated FS Semantics which caused put operation to fail.
> bash-4.2$ ozone fs -ls ofs://om/s3v/bucketfso Found 2 items -rw-rw-rw- 1
> hadoop hadoop 0 2022-12-05 22:34 ofs://om/s3v/bucketfso/test1
> {code}
> File is created _when_ a folder is expected. If it were correctly set it
> would result in seeing _*drwxrwxrwx*_ for the test1 key as in,
> {code:java}
> drwxrwxrwx - hadoop hadoop 0 2022-12-05 22:40
> ofs://om/s3v/bucketfos/test1
> {code}
>
> In contrast, the same aws command issued on a *OBJECT_STORE* layout bucket
> results in :
> {code:java}
> bash-4.2$ ozone sh key list /s3v/bucketobs
> [ {
> "volumeName" : "s3v",
> "bucketName" : "bucketobs",
> "name" : "test1",
> "dataSize" : 0,
> "creationTime" : "2022-12-05T22:33:21.737Z",
> "modificationTime" : "2022-12-05T22:33:21.746Z",
> "replicationConfig" : {
> "replicationFactor" : "ONE",
> "requiredNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { }
> }, {
> "volumeName" : "s3v",
> "bucketName" : "bucketobs",
> "name" : "test1/README.md",
> "dataSize" : 3879,
> "creationTime" : "2022-12-05T22:34:02.402Z",
> "modificationTime" : "2022-12-05T22:37:00.758Z",
> "replicationConfig" : {
> "replicationFactor" : "ONE",
> "requiredNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { }
> } ]{code}
> As is expected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
