[GitHub] [ozone] xichen01 opened a new pull request, #4127: HDDS-7710. Support AWS s3 ListObjects API's encodingType request parameter

[GitBox]

xichen01 opened a new pull request, #4127:
URL: https://github.com/apache/ozone/pull/4127

   ## What changes were proposed in this pull request?
   Support AWS s3 ListObjects API's encodingType request parameter
   
   refer:
   
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType
   
   
   current Ozone will encode `key` and `prefix` in the `ListObjects` response 
in the default, however, the AWS s3 has a request parameter of `encoding-type` 
that can control whether to encode fields such as key in the response 
   
   ### Why need add this feature
   The Ozone default encoding key can cause some bugs
   For example, when using go-sdk `ListObjectsV2`, the encoded key is returned
   
   this is the example code:
   you can put some keys that contain special characters in the key name (such 
as "="), execute this code and check the return value, you will find that 
special characters are encoded. (If using AWS s3 key name will be returned 
correctly)
   ```golang
   package main
   
   import (
        "fmt"
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/s3"
   )
   
   const (
        AccessKeyID     = "id"
        SecretAccessKey = "key"
        BucketRegion    = "none"
        BucketName      = "bucket1"
        Endpoint        = "http://localhost:9878";
   )
   
   func ListItems(client *s3.S3, bucketName string, prefix string) 
(*s3.ListObjectsV2Output, error) {
        res, err := client.ListObjectsV2(&s3.ListObjectsV2Input{
                Bucket: aws.String(bucketName),
                Prefix: aws.String(prefix),
        })
        if err != nil {
                return nil, err
        }
        return res, nil
   }
   
   func main() {
        // create session
        sess := session.Must(session.NewSession(aws.NewConfig().
                WithRegion(BucketRegion).
                WithCredentials(credentials.NewStaticCredentials(AccessKeyID, 
SecretAccessKey, "")).
                WithEndpoint(Endpoint).
                WithS3ForcePathStyle(true)))
        s3session := s3.New(sess)
        prefixName := ""
   
        bucketObjects, err := ListItems(s3session, BucketName, prefixName)
        if err != nil {
                fmt.Printf("Couldn't retrieve bucket items: %v", err)
                return
        }
   
        for _, item := range bucketObjects.Contents {
                fmt.Printf("Name: %s, Last Modified: %s\n", *item.Key, 
*item.LastModified)
        }
   }
   ```
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-7710
   
   ## How was this patch tested?
   
   
   - create a bucket and put some keys 
   ```shell
   [root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh bucket create 
s3v/bucket1
   [root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh key put 
/s3v/bucket1/file=124 ~/testfile
   [root@Linux /root/ozone-1.3.0-SNAPSHOT]% bin/ozone sh key put 
/s3v/bucket1/file==124 ~/testfile
   [root@Linux /root/ozone-1.3.0-SNAPSHOT]
   ```
   
   - Prepare a postman APP
   - configure the postman APP's S3 Authorization
   <img width="1260" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209475686-a2840e47-f494-4a41-8610-16d05bd00b46.png";>
   
   
   - Create a `ListObjects` request using postman as shown below
   
   If you specify the encoding-type request parameter, S3 includes this element 
in the response, and returns encoded key name values in the following response 
elements:
   
   Get URL: 
`http://s3.localhost:9878/bucket1/?encoding-type=url&prefix=file=&start-after=file=&delimiter==&list-type=2`
   <img width="974" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209475244-e89e151a-bcce-4dea-98ed-3f2d687e4fae.png";>
   
   - Send the `ListObjects` request again, but without the `encodingtype` 
parameter
   
   
   If you do not specify the encoding-type request parameter, Amazon S3 does 
not include this element in the response, and returns unencoded key name values 
in the following response elements:
   
   Get URL: 
`http://s3.localhost:9878/bucket1/?prefix=file=&start-after=file=&delimiter==&list-type=2`
   <img width="988" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209475288-f76e4e46-d7e2-489f-b929-3bd121ecacf1.png";>
   
   - if the encoding-type request parameter is invalid
   <img width="913" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209475831-020d681f-16a3-41af-a3bb-bd74117e4a47.png";>
   
   
   
   
   
   ### refer AWS s3's response:
   
   I verified the AWS response using the postman
   
   the key in the bucket
   ```shell
   [root@VM-8-3-centos ~]$ aws s3 ls s3://pony-bucket1/
   2022-12-23 15:08:39       2884 file=124
   2022-12-23 15:19:23        234 file==hosts
   [root@VM-8-3-centos ~]$
   ```
   
   If you specify the encoding-type request parameter, Amazon S3 includes this 
element in the response, and returns encoded key name values in the following 
response elements:
   
   `Delimiter`, `Prefix`, `Key`, and `StartAfter`.
   
   <img width="1201" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209474637-16961931-deb6-4924-b1e2-ec177e1e3629.png";>
   
   If you do not specify the encoding-type request parameter, Amazon S3 does 
not include this element in the response, and returns unencoded key name values 
in the following response elements:
   
   <img width="1003" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209474508-01d9898a-2436-42b4-8277-b1d3055fb60d.png";>
   
   if the encoding-type request parameter is invalid
   <img width="929" alt="image" 
src="https://user-images.githubusercontent.com/32928346/209475771-fe2c4641-523e-4181-a7e5-1d98419fb3bd.png";>
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org