[
https://issues.apache.org/jira/browse/HDDS-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644044#comment-17644044
]
Ethan Rose commented on HDDS-7594:
----------------------------------
Hi [~NeilJoshi] this looks like expected behavior to me. I'm a bit confused by
the issue description.
{quote}With FSO bucket layout buckets, folders created with the s3-gateway
create 0 byte files on file system.
{quote}
S3 gateway exposes the S3 API which does not support directory creation. I'm
not sure what it means to create a directory from S3 gateway.
{quote}Unlike Object Store bucket layout buckets that create folders correctly
when created through the s3-gateway, folder creation on FSO buckets with s3
gateway results in 0 byte files
{quote}
OBS buckets also do not have the concept of directories. What it seems you are
observing here is the zero byte key {{test1}} not conflicting with the key
{{test1/README.md}} due to the OBS bucket layout.
{quote}This has caused problems with interoperability for applications
accessing ozone through both ofs and the s3 gateway, such as Trino.
{quote}
Can you clarify the use case here? If Trino is talking to Ozone through s3
gateway is should be expecting to run on s3 itself, or s3 compatible storage,
neither of which support directories. Are Trino or other engines actually doing
things like {{put key1}} followed by {{put key1/key2}} when running on S3? If
we want to support this I think we would need a client side config (for ozone
client on s3 gateway) to convert 0 byte file writes from S3 API to directory
creates if the underlying bucket type is FSO. I don't think we would want to do
this on OM or it would affect all clients even outside of S3.
> [FSO] Folders created through S3G are created on file system as "files".
> ------------------------------------------------------------------------
>
> Key: HDDS-7594
> URL: https://issues.apache.org/jira/browse/HDDS-7594
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Neil Joshi
> Priority: Major
>
> With FSO bucket layout buckets, folders created with the s3-gateway create 0
> byte files on file system. Unlike Object Store bucket layout buckets that
> create folders correctly when created through the s3-gateway, folder creation
> on FSO buckets with s3 gateway results in 0 byte files. Expected is the same
> behaviour. This has caused problems with interoperability for applications
> accessing ozone through both ofs and the s3 gateway, such as Trino.
>
> Problem can be produced issuing aws-cli folder creation to the s3 gateway on
> an ozone cluster layout {*}FILE_SYSTEM_OPTIMIZED{*}:
>
> {code:java}
> bash-4.2$ ozone sh bucket create --layout=FILE_SYSTEM_OPTIMIZED /s3v/bucketfso
> ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object
> --bucket bucketfso --key test1
> ozone$ aws s3api --profile ozone --endpoint http://localhost:9878 put-object
> --bucket bucketfso --key test1/README.md --body ./README.md
> An error occurred (InvalidRequest) when calling the PutObject operation: An
> error occurred (InvalidRequest) when calling the PutObject/MPU PartUpload
> operation: ozone.om.enable.filesystem.paths is enabled Keys are considered as
> Unix Paths. Path has Violated FS Semantics which caused put operation to fail.
> bash-4.2$ ozone fs -ls ofs://om/s3v/bucketfso Found 2 items -rw-rw-rw- 1
> hadoop hadoop 0 2022-12-05 22:34 ofs://om/s3v/bucketfso/test1
> {code}
> File is created _when_ a folder is expected. If it were correctly set it
> would result in seeing _*drwxrwxrwx*_ for the test1 key as in,
> {code:java}
> drwxrwxrwx - hadoop hadoop 0 2022-12-05 22:40
> ofs://om/s3v/bucketfos/test1
> {code}
>
> In contrast, the same aws command issued on a *OBJECT_STORE* layout bucket
> results in :
> {code:java}
> bash-4.2$ ozone sh key list /s3v/bucketobs
> [ {
> "volumeName" : "s3v",
> "bucketName" : "bucketobs",
> "name" : "test1",
> "dataSize" : 0,
> "creationTime" : "2022-12-05T22:33:21.737Z",
> "modificationTime" : "2022-12-05T22:33:21.746Z",
> "replicationConfig" : {
> "replicationFactor" : "ONE",
> "requiredNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { }
> }, {
> "volumeName" : "s3v",
> "bucketName" : "bucketobs",
> "name" : "test1/README.md",
> "dataSize" : 3879,
> "creationTime" : "2022-12-05T22:34:02.402Z",
> "modificationTime" : "2022-12-05T22:37:00.758Z",
> "replicationConfig" : {
> "replicationFactor" : "ONE",
> "requiredNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { }
> } ]{code}
> As is expected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
