Mladjan Gadzic created HDDS-9229:
------------------------------------
Summary: [FSO] S3A compatibility - dfs -put creates dir and a file
Key: HDDS-9229
URL: https://issues.apache.org/jira/browse/HDDS-9229
Project: Apache Ozone
Issue Type: Bug
Reporter: Mladjan Gadzic
Bucket with FSO layout (i.e. /s3v/fso) is created on Ozone side (unsecured
Ozone Docker cluster).
On Hadoop side (official 3.3.4 release) hdfs dfs -put command is issued (i.e
README.txt file to be put to s3a://fso/).
What happens is: 2 keys are created (dir /fso/README.txt/ and file
README.txt/README.txt._COPYING_)
Expected behavior: 1 key is created, file /fso/README.txt
Logs from Ozone:
{code:java}
bash-4.2$ ozone sh key list /s3v/fso
[ {
"volumeName" : "s3v",
"bucketName" : "fso",
"name" : "README.txt/",
"dataSize" : 0,
"creationTime" : "2023-08-30T14:56:49.675Z",
"modificationTime" : "2023-08-30T14:56:49.675Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
},
"metadata" : { },
"file" : false
}, {
"volumeName" : "s3v",
"bucketName" : "fso",
"name" : "README.txt/EADME.txt._COPYING_",
"dataSize" : 175,
"creationTime" : "2023-08-30T14:56:49.675Z",
"modificationTime" : "2023-08-30T14:56:50.295Z",
"replicationConfig" : {
"replicationFactor" : "ONE",
"requiredNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"file" : true
} ]{code}
Logs from Hadoop:
{code:java}
bin/hdfs dfs -Dfs.s3a.access.key=1 -Dfs.s3a.secret.key=1
-Dfs.s3a.endpoint=http://localhost:9878 -Dfs.s3a.path.style.access=true -put
README.txt s3a://fso/
2023-08-30 14:56:41,596 INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2023-08-30 14:56:41,747 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2023-08-30 14:56:41,747 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system started
2023-08-30 14:56:43,663 INFO impl.DirectoryPolicyImpl: Directory markers will
be kept
2023-08-30 14:56:50,503 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
metrics system...
2023-08-30 14:56:50,503 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system stopped.
2023-08-30 14:56:50,504 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system shutdown complete.{code}
Hadoop configuration (core-site.xml)
{code:java}
<configuration>
<property>
<name>fs.s3a.access.key</name>
<value>1</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>1</value>
</property><property>
<name>fs.s3a.capability.directory.marker.policy.delete</name>
<value>keep</value>
</property>
<property>
<name>fs.s3a.capability.directory.marker.action.delete</name>
<value>keep</value>
</property>
<property>
<name>fs.s3a.directory.marker.retention</name>
<value>keep</value>
</property>
</configuration> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
