[
https://issues.apache.org/jira/browse/HDDS-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
UENISHI Kota updated HDDS-7992:
-------------------------------
Description:
AWSCLI v2 running command "{{aws s3
--endpoint=[https://our-endpoint.example.com/] cp s3://some/object
s3://some/destination}}" fails with returning huge error message printing the
object itself.
The "{{--debug}}" log indicates that awscli v2 sends out weird HTTP request
beforehand:
{quote} 2023-02-21 11:15:17,570 - ThreadPoolExecutor-1_0 - botocore.endpoint -
DEBUG - Sending http request: <AWSPreparedRequest stream_output=False,
method=GET, url=https://endpoint.example.com/kota/32MiB?tagging,
headers=\{'User-Agent': b'aws-cli/2.9.15 Python/3.9.11 Linux/5.15.0-56-generic
exe/x86_64.ubuntu.22 prompt/off command/s3.cp', 'X-Amz-Date':
b'20230221T021517Z', 'X-Amz-Content-SHA256':
b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
'Authorization': b'AWS4-HMAC-SHA256
[email protected]/20230221/us-east-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=dd3b6b6e19776e123c81c699b1f45825507857c19e817d5a823fca3aa7361aea'}>
2023-02-21 11:15:17,571 - ThreadPoolExecutor-1_0 - botocore.httpsession - DEBUG
- Certificate path: /usr/local/aws-cli/v2/2.9.15/dist/awscli/botocore/cacert.pem
2023-02-21 11:15:17,593 - ThreadPoolExecutor-1_0 - urllib3.connectionpool -
DEBUG - https://endpoint.example.com:443 "GET /kota/32MiB?tagging HTTP/1.1" 200
33554432
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
Response headers: \{'Date': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Cache-Control':
'no-cache', 'Expires': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Pragma': 'no-cache',
'Content-Type': 'application/octet-stream', 'X-Content-Type-Options':
'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-FRAME-OPTIONS':
'SAMEORIGIN', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Tue, 21 Feb 2023
01:57:51 GMT', 'Server': 'Ozone', 'x-amz-id-2': '5Q3e2Cl7', 'x-amz-request-id':
'e8c7b5a0-d0fe-4cfa-8508-abd34574c793', 'Content-Length': '33554432'\}
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
Response body:
b'tm\x1dv3\xf7\x880=\x94\x17\x14\xd1Y\xa7\xa3\x11\xa8\x04%\xa8\xb2\x19Q\x98G\xde\xca/,\x04\x0e\xfe\x18\xcdY\xf4Pb\xfa\xc3\x8fy%\x00GB-l\x14\xab:\x0f\xeb\xf1\xbb\\/\x10\x8em\x19<W\xc7\xc7+0\xd2.\x06\x9at\x01\xc8Y\xda\xb4Y!n\x0c`>\x88\x14\xea\xf4\x9e\xd4\x90\x8e\xc8\xf3\x92e\x9c+W\xd5\x04\x9d\xf6\xa3\xa7c\xd9\x17W\xf2\x88
{quote}
The debug log illustrates that AWSCLI is trying to reach out [GetObjectTagging
API |https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html]
available in AWS S3, which is not available in Apache Ozone.
The workaround for this issue is to use AWSCLI v1.
The solution for this issue would be two-level:
# Handle unknown get parameters as as unknown in S3Gateway and return some
error like 500 or whatever. Pros: easy. Cons: not easy? to list up all 'known'
get parameters.
# Implement tagging API. Pros: right fix. Cons: needs protobuf-format change of
Key metadata. Maybe adding some map data structure for user-defined metadata.
was:
AWSCLI v2 running command "{{aws s3
--endpoint=[https://our-endpoint.example.com/] cp s3://some/object
s3://some/destination}}" fails with returning huge error message printing the
object itself.
The "{{--debug}}" log indicates that awscli v2 sends out weird HTTP request
beforehand:
{quote} 2023-02-21 11:15:17,570 - ThreadPoolExecutor-1_0 - botocore.endpoint -
DEBUG - Sending http request: <AWSPreparedRequest stream_output=False,
method=GET, url=https://endpoint.example.com/kota/32MiB?tagging,
headers=\{'User-Agent': b'aws-cli/2.9.15 Python/3.9.11 Linux/5.15.0-56-generic
exe/x86_64.ubuntu.22 prompt/off command/s3.cp', 'X-Amz-Date':
b'20230221T021517Z', 'X-Amz-Content-SHA256':
b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
'Authorization': b'AWS4-HMAC-SHA256
[email protected]/20230221/us-east-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=dd3b6b6e19776e123c81c699b1f45825507857c19e817d5a823fca3aa7361aea'}>
2023-02-21 11:15:17,571 - ThreadPoolExecutor-1_0 - botocore.httpsession - DEBUG
- Certificate path: /usr/local/aws-cli/v2/2.9.15/dist/awscli/botocore/cacert.pem
2023-02-21 11:15:17,593 - ThreadPoolExecutor-1_0 - urllib3.connectionpool -
DEBUG - https://endpoint.example.com:443 "GET /kota/32MiB?tagging HTTP/1.1" 200
33554432
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
Response headers: \{'Date': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Cache-Control':
'no-cache', 'Expires': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Pragma': 'no-cache',
'Content-Type': 'application/octet-stream', 'X-Content-Type-Options':
'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-FRAME-OPTIONS':
'SAMEORIGIN', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Tue, 21 Feb 2023
01:57:51 GMT', 'Server': 'Ozone', 'x-amz-id-2': '5Q3e2Cl7', 'x-amz-request-id':
'e8c7b5a0-d0fe-4cfa-8508-abd34574c793', 'Content-Length': '33554432'\}
2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
Response body:
b'tm\x1dv3\xf7\x880=\x94\x17\x14\xd1Y\xa7\xa3\x11\xa8\x04%\xa8\xb2\x19Q\x98G\xde\xca/,\x04\x0e\xfe\x18\xcdY\xf4Pb\xfa\xc3\x8fy%\x00GB-l\x14\xab:\x0f\xeb\xf1\xbb\\/\x10\x8em\x19<W\xc7\xc7+0\xd2.\x06\x9at\x01\xc8Y\xda\xb4Y!n\x0c`>\x88\x14\xea\xf4\x9e\xd4\x90\x8e\xc8\xf3\x92e\x9c+W\xd5\x04\x9d\xf6\xa3\xa7c\xd9\x17W\xf2\x88
{quote}
The debug log illustrates that AWSCLI is trying to reach out [GetObjectTagging
API |https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html]
available in AWS S3, which is not available in Apache Ozone.
The solution for this issue would be two-level:
# Handle unknown get parameters as as unknown in S3Gateway and return some
error like 500 or whatever. Pros: easy. Cons: not easy? to list up all 'known'
get parameters.
# Implement tagging API. Pros: right fix. Cons: needs protobuf-format change of
Key metadata. Maybe adding some map data structure for user-defined metadata.
> Copy and move command for MPU objects from awscli v2 not working
> ----------------------------------------------------------------
>
> Key: HDDS-7992
> URL: https://issues.apache.org/jira/browse/HDDS-7992
> Project: Apache Ozone
> Issue Type: Bug
> Components: S3
> Affects Versions: 1.3.0, 1.2.1
> Environment: * Affected AWSCLI version: aws-cli/2.10.1 Python/3.9.11
> Linux/6.1.12-arch1-1 docker/x86_64.amzn.2 prompt/off
> * It does not happen for AWSCLI v1. So our workaround is to use it for now.
> * We use Ubuntu Linux 22.04 for client side
> * We have two clusters Running Ozone 1.3, one runs Ubuntu 18.04 and the
> other runs 22.04.
> Reporter: UENISHI Kota
> Priority: Major
>
> AWSCLI v2 running command "{{aws s3
> --endpoint=[https://our-endpoint.example.com/] cp s3://some/object
> s3://some/destination}}" fails with returning huge error message printing the
> object itself.
> The "{{--debug}}" log indicates that awscli v2 sends out weird HTTP request
> beforehand:
> {quote} 2023-02-21 11:15:17,570 - ThreadPoolExecutor-1_0 - botocore.endpoint
> - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False,
> method=GET, url=https://endpoint.example.com/kota/32MiB?tagging,
> headers=\{'User-Agent': b'aws-cli/2.9.15 Python/3.9.11
> Linux/5.15.0-56-generic exe/x86_64.ubuntu.22 prompt/off command/s3.cp',
> 'X-Amz-Date': b'20230221T021517Z', 'X-Amz-Content-SHA256':
> b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
> 'Authorization': b'AWS4-HMAC-SHA256
> [email protected]/20230221/us-east-1/s3/aws4_request,
> SignedHeaders=host;x-amz-content-sha256;x-amz-date,
> Signature=dd3b6b6e19776e123c81c699b1f45825507857c19e817d5a823fca3aa7361aea'}>
> 2023-02-21 11:15:17,571 - ThreadPoolExecutor-1_0 - botocore.httpsession -
> DEBUG - Certificate path:
> /usr/local/aws-cli/v2/2.9.15/dist/awscli/botocore/cacert.pem
> 2023-02-21 11:15:17,593 - ThreadPoolExecutor-1_0 - urllib3.connectionpool -
> DEBUG - https://endpoint.example.com:443 "GET /kota/32MiB?tagging HTTP/1.1"
> 200 33554432
> 2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
> Response headers: \{'Date': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Cache-Control':
> 'no-cache', 'Expires': 'Tue, 21 Feb 2023 02:15:17 GMT', 'Pragma': 'no-cache',
> 'Content-Type': 'application/octet-stream', 'X-Content-Type-Options':
> 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-FRAME-OPTIONS':
> 'SAMEORIGIN', 'Accept-Ranges': 'bytes', 'Last-Modified': 'Tue, 21 Feb 2023
> 01:57:51 GMT', 'Server': 'Ozone', 'x-amz-id-2': '5Q3e2Cl7',
> 'x-amz-request-id': 'e8c7b5a0-d0fe-4cfa-8508-abd34574c793', 'Content-Length':
> '33554432'\}
> 2023-02-21 11:15:17,945 - ThreadPoolExecutor-1_0 - botocore.parsers - DEBUG -
> Response body:
> b'tm\x1dv3\xf7\x880=\x94\x17\x14\xd1Y\xa7\xa3\x11\xa8\x04%\xa8\xb2\x19Q\x98G\xde\xca/,\x04\x0e\xfe\x18\xcdY\xf4Pb\xfa\xc3\x8fy%\x00GB-l\x14\xab:\x0f\xeb\xf1\xbb\\/\x10\x8em\x19<W\xc7\xc7+0\xd2.\x06\x9at\x01\xc8Y\xda\xb4Y!n\x0c`>\x88\x14\xea\xf4\x9e\xd4\x90\x8e\xc8\xf3\x92e\x9c+W\xd5\x04\x9d\xf6\xa3\xa7c\xd9\x17W\xf2\x88
> {quote}
> The debug log illustrates that AWSCLI is trying to reach out
> [GetObjectTagging API
> |https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html]
> available in AWS S3, which is not available in Apache Ozone.
> The workaround for this issue is to use AWSCLI v1.
> The solution for this issue would be two-level:
> # Handle unknown get parameters as as unknown in S3Gateway and return some
> error like 500 or whatever. Pros: easy. Cons: not easy? to list up all
> 'known' get parameters.
> # Implement tagging API. Pros: right fix. Cons: needs protobuf-format change
> of Key metadata. Maybe adding some map data structure for user-defined
> metadata.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
