Aleksei Ieshin created HDDS-17303:
-------------------------------------
Summary: Allow enforcing bucket replication over the S3 storage
class header
Key: HDDS-17303
URL: https://issues.apache.org/jira/browse/HDDS-17303
Project: Apache Ozone
Issue Type: Improvement
Components: OM, S3
Reporter: Aleksei Ieshin
Fix For: 2.2.1, 2.2.0, 2.1.0
h3. Problem
Before HDDS-10979 (2.1.0), the S3 Gateway gave an EC bucket precedence over the
client's {{x-amz-storage-class}} header ([S3Utils.java#L85-L97 in
2.0.0|https://github.com/apache/ozone/blob/ozone-2.0.0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java#L85-L97]).
The comment there gave the reason: "keys created from file system under EC
bucket will always be EC'd."
Since HDDS-10979, any non-empty header wins outright ([S3Utils.java#L89-L90 in
2.2.1|https://github.com/apache/ozone/blob/ozone-2.2.1/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java#L89-L90]).
{{STANDARD}} now always means RATIS/THREE, even in a bucket the administrator
configured as EC. Setting EC on the bucket does not help, because the header
still wins.
The same override exists outside S3:
{{OzoneConfigUtil.resolveReplicationConfigPreference}} uses any client-supplied
replication before the bucket default ([OzoneConfigUtil.java#L44-L49 in
2.2.1|https://github.com/apache/ozone/blob/ozone-2.2.1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneConfigUtil.java#L44-L49]).
h3. Impact (observed in production)
We run an EC-by-default cluster. After upgrading from 2.0.0 to 2.2.1, a bulk
copy job used an off-the-shelf S3 copy tool that sends {{{}x-amz-storage-class:
STANDARD{}}}. All of its keys were written RATIS/THREE instead of EC:
* about 3x the raw capacity for that data, instead of 1.67x with rs-3-2;
* on our configuration, the first use of Ratis DataStream on the datanodes,
with a large jump in threads and file descriptors per datanode;
* no error or warning. We found it in the OM audit log ({{{}replicationType:
RATIS{}}}) only because datanode resource graphs moved.
The only fix today is to change every client, and cluster operators often don't
control the clients.
The storage policy work (HDDS-11233, HDDS-16039) follows the same precedence:
request, then bucket, then default. Once it lands, a client sending {{GLACIER}}
could place data on COLD storage in a bucket the administrator set to HOT.
h3. Proposal
Add an operator switch, *off by default* so current behaviour is unchanged,
that makes the server-side settings authoritative. For example
{{ozone.s3g.storage-class.client-override.enabled}} (default {{{}true{}}}).
When it is {{{}false{}}}:
* The S3 Gateway ignores {{x-amz-storage-class}} and
{{x-amz-meta-storage-config}} when choosing replication, and sends none to the
OM. The bucket default applies, and then the server default.
* Once storage policies exist, the same switch also stops the header from
choosing the storage policy, so the bucket decides both.
* The header is still validated: unknown classes are still rejected with
{{{}InvalidStorageClass{}}}, so clients see the same errors. Valid classes are
accepted and {*}ignored{*}, not rejected, so tools that always send
{{STANDARD}} keep working.
* {{StorageClass}} on GET, HEAD and LIST keeps reporting the class derived
from the key's actual replication (HDDS-12877), so clients can see what they
got.
{{S3StorageClassResolver}} from HDDS-16039 looks like the natural place for the
switch.
Open questions:
# Should the switch apply to all writes (bucket default, else server default),
or only to buckets with an explicit replication setting? The second is more
conservative, since an administrator opts in per bucket by setting replication.
# One switch for replication and storage policy, or two?
h3. Possible follow-up (OM level)
A bucket-level "enforce default replication" attribute, or an OM-wide setting,
applied in {{{}resolveReplicationConfigPreference{}}}. It would cover every
client path ({{{}ofs{}}}, native clients with {{ozone.replication}} set), not
only S3. It is a larger change, so we suggest a separate sub-task under
HDDS-10980.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
