[
https://issues.apache.org/jira/browse/HADOOP-12020?focusedWorklogId=766160&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-766160
]
ASF GitHub Bot logged work on HADOOP-12020:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/May/22 16:39
Start Date: 04/May/22 16:39
Worklog Time Spent: 10m
Work Description: monthonk commented on code in PR #3877:
URL: https://github.com/apache/hadoop/pull/3877#discussion_r865045738
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AStorageClass.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.nio.file.AccessDeniedException;
+import java.util.Map;
+
+import com.amazonaws.services.s3.model.StorageClass;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.contract.s3a.S3AContract;
+import org.apache.hadoop.test.GenericTestUtils;
+
+import static org.apache.hadoop.fs.s3a.Constants.STORAGE_CLASS;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.disableFilesystemCaching;
+import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.HeaderProcessing.XA_STORAGE_CLASS;
+import static org.apache.hadoop.fs.s3a.impl.HeaderProcessing.decodeBytes;
+
+/**
+ * Tests of storage class.
+ */
+public class ITestS3AStorageClass extends AbstractS3ATestBase {
+
+ @Override
+ protected Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(conf, STORAGE_CLASS);
+
+ return conf;
+ }
+
+ /*
+ * This test ensures the default storage class configuration (no config or
null)
+ * works well with create and copy operations
+ */
+ @Test
+ public void testCreateAndCopyObjectWithStorageClassDefault() throws
Throwable {
+ Configuration conf = this.createConfiguration();
+ S3AContract contract = (S3AContract) createContract(conf);
+ contract.init();
+
+ FileSystem fs = contract.getTestFileSystem();
+ Path dir = methodPath();
+ fs.mkdirs(dir);
+ assertObjectHasNoStorageClass(dir);
+ Path path = new Path(dir, "file1");
+ ContractTestUtils.touch(fs, path);
+ assertObjectHasNoStorageClass(path);
+ Path path2 = new Path(dir, "file1");
+ fs.rename(path, path2);
+ assertObjectHasNoStorageClass(path2);
+ }
+
+ /*
+ * Verify object can be created and copied correctly
+ * with specified storage class
+ */
+ @Test
+ public void testCreateAndCopyObjectWithStorageClassReducedRedundancy()
throws Throwable {
+ Configuration conf = this.createConfiguration();
+ conf.set(STORAGE_CLASS, StorageClass.ReducedRedundancy.toString());
Review Comment:
i don't mind adding them in Constants file, but we will have to keep it up
to date with aws-sdk. i wonder if it's ok?
Issue Time Tracking
-------------------
Worklog Id: (was: 766160)
Time Spent: 40m (was: 0.5h)
> Support AWS S3 reduced redundancy storage class
> -----------------------------------------------
>
> Key: HADOOP-12020
> URL: https://issues.apache.org/jira/browse/HADOOP-12020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Environment: Hadoop on AWS
> Reporter: Yann Landrin-Schweitzer
> Assignee: Monthon Klongklaew
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Amazon S3 uses, by default, the NORMAL_STORAGE class for s3 objects.
> This offers, according to Amazon's material, 99.99999999% reliability.
> For many applications, however, the 99.99% reliability offered by the
> REDUCED_REDUNDANCY storage class is amply sufficient, and comes with a
> significant cost saving.
> HDFS, when using the legacy s3n protocol, or the new s3a scheme, should
> support overriding the default storage class of created s3 objects so that
> users can take advantage of this cost benefit.
> This would require minor changes of the s3n and s3a drivers, using
> a configuration property fs.s3n.storage.class to override the default storage
> when desirable.
> This override could be implemented in Jets3tNativeFileSystemStore with:
> S3Object object = new S3Object(key);
> ...
> if(storageClass!=null) object.setStorageClass(storageClass);
> It would take a more complex form in s3a, e.g. setting:
> InitiateMultipartUploadRequest initiateMPURequest =
> new InitiateMultipartUploadRequest(bucket, key, om);
> if(storageClass !=null ) {
> initiateMPURequest =
> initiateMPURequest.withStorageClass(storageClass);
> }
> and similar statements in various places.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
