[
https://issues.apache.org/jira/browse/HDDS-17303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130205#comment-18130205
]
Aleksei Ieshin commented on HDDS-17303:
---------------------------------------
I think some implementation details might be leveraged from
https://issues.apache.org/jira/browse/HDDS-15087
> Allow enforcing bucket replication over the S3 storage class header
> -------------------------------------------------------------------
>
> Key: HDDS-17303
> URL: https://issues.apache.org/jira/browse/HDDS-17303
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM, S3
> Reporter: Aleksei Ieshin
> Priority: Major
> Fix For: 2.1.0, 2.2.0, 2.2.1
>
>
> h3. Problem
> Before HDDS-10979 (2.1.0), the S3 Gateway gave an EC bucket precedence over
> the client's {{x-amz-storage-class}} header ([S3Utils.java#L85-L97 in
> 2.0.0|https://github.com/apache/ozone/blob/ozone-2.0.0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java#L85-L97]).
> The comment there gave the reason: "keys created from file system under EC
> bucket will always be EC'd."
> Since HDDS-10979, any non-empty header wins outright ([S3Utils.java#L89-L90
> in
> 2.2.1|https://github.com/apache/ozone/blob/ozone-2.2.1/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java#L89-L90]).
> {{STANDARD}} now always means RATIS/THREE, even in a bucket the
> administrator configured as EC. Setting EC on the bucket does not help,
> because the header still wins.
> The same override exists outside S3:
> {{OzoneConfigUtil.resolveReplicationConfigPreference}} uses any
> client-supplied replication before the bucket default
> ([OzoneConfigUtil.java#L44-L49 in
> 2.2.1|https://github.com/apache/ozone/blob/ozone-2.2.1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneConfigUtil.java#L44-L49]).
> h3. Impact (observed in production)
> We run an EC-by-default cluster. After upgrading from 2.0.0 to 2.2.1, a bulk
> copy job used an off-the-shelf S3 copy tool that sends
> {{{}x-amz-storage-class: STANDARD{}}}. All of its keys were written
> RATIS/THREE instead of EC:
> * about 3x the raw capacity for that data, instead of 1.67x with rs-3-2;
> * on our configuration, the first use of Ratis DataStream on the datanodes,
> with a large jump in threads and file descriptors per datanode;
> * no error or warning. We found it in the OM audit log ({{{}replicationType:
> RATIS{}}}) only because datanode resource graphs moved.
> The only fix today is to change every client, and cluster operators often
> don't control the clients.
> The storage policy work (HDDS-11233, HDDS-16039) follows the same precedence:
> request, then bucket, then default. Once it lands, a client sending
> {{GLACIER}} could place data on COLD storage in a bucket the administrator
> set to HOT.
> h3. Proposal
> Add an operator switch, *off by default* so current behaviour is unchanged,
> that makes the server-side settings authoritative. For example
> {{ozone.s3g.storage-class.client-override.enabled}} (default {{{}true{}}}).
> When it is {{{}false{}}}:
> * The S3 Gateway ignores {{x-amz-storage-class}} and
> {{x-amz-meta-storage-config}} when choosing replication, and sends none to
> the OM. The bucket default applies, and then the server default.
> * Once storage policies exist, the same switch also stops the header from
> choosing the storage policy, so the bucket decides both.
> * The header is still validated: unknown classes are still rejected with
> {{{}InvalidStorageClass{}}}, so clients see the same errors. Valid classes
> are accepted and {*}ignored{*}, not rejected, so tools that always send
> {{STANDARD}} keep working.
> * {{StorageClass}} on GET, HEAD and LIST keeps reporting the class derived
> from the key's actual replication (HDDS-12877), so clients can see what they
> got.
> {{S3StorageClassResolver}} from HDDS-16039 looks like the natural place for
> the switch.
> Open questions:
> # Should the switch apply to all writes (bucket default, else server
> default), or only to buckets with an explicit replication setting? The second
> is more conservative, since an administrator opts in per bucket by setting
> replication.
> # One switch for replication and storage policy, or two?
> h3. Possible follow-up (OM level)
> A bucket-level "enforce default replication" attribute, or an OM-wide
> setting, applied in {{{}resolveReplicationConfigPreference{}}}. It would
> cover every client path ({{{}ofs{}}}, native clients with
> {{ozone.replication}} set), not only S3. It is a larger change, so we suggest
> a separate sub-task under HDDS-10980.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
