[
https://issues.apache.org/jira/browse/HDDS-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688705#comment-16688705
]
Bharat Viswanadham edited comment on HDDS-791 at 11/15/18 9:59 PM:
-------------------------------------------------------------------
Smoke test Output ran locally with S3Gateway endpoint and aws s3 endpoint.
region:us-east-2
{code:java}
HW13865:smoketest bviswanadham$ robot -v OZONE_S3_SET_CREDENTIALS:false -v
OZONE_TEST:false -v BUCKET:b1-1992 -v DESTBUCKET:b1-1993 -v
ENDPOINT_URL:https://s3.us-east-2.amazonaws.com s3/objectputget.robot
==============================================================================
Objectputget :: S3 gateway test with aws cli
==============================================================================
Put object to s3 | PASS |
------------------------------------------------------------------------------
Get object from s3 ..sh:
md5sum: command not found
.sh: md5sum: command not found
Get object from s3 | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes | PASS |
------------------------------------------------------------------------------
Incorrect values for end and start offset | PASS |
------------------------------------------------------------------------------
Zero byte file | PASS |
------------------------------------------------------------------------------
Objectputget :: S3 gateway test with aws cli | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
Output:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/output.xml
Log:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/log.html
Report:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/report.html
HW13865:smoketest bviswanadham$ ./test.sh --env ozones3 s3/objectputget.robot
-------------------------------------------------
Executing test(s): [s3/objectputget.robot]
Cluster type: ozones3
Compose file:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/../compose/ozones3/docker-compose.yaml
Output dir:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/result
Command to rerun: ./test.sh --keep --env ozones3 s3/objectputget.robot
-------------------------------------------------
Removing network ozones3_default
WARNING: Network ozones3_default not found.
Creating network "ozones3_default" with the default driver
Creating ozones3_datanode_1 ...
Creating ozones3_scm_1 ...
Creating ozones3_ozoneManager_1 ...
Creating ozones3_s3g_1 ...
Creating ozones3_scm_1
Creating ozones3_datanode_1
Creating ozones3_s3g_1
Creating ozones3_ozoneManager_1 ... done
Waiting 30s for cluster start up...
==============================================================================
Objectputget :: S3 gateway test with aws cli
==============================================================================
Put object to s3 | PASS |
------------------------------------------------------------------------------
Get object from s3 | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes | PASS |
------------------------------------------------------------------------------
Incorrect values for end and start offset | PASS |
------------------------------------------------------------------------------
Zero byte file | PASS |
------------------------------------------------------------------------------
Objectputget :: S3 gateway test with aws cli | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
Output: /opt/hadoop/smoketest/result/robot-ozones3-s3_objectputget.robot.xml
Stopping ozones3_ozoneManager_1 ... done
Stopping ozones3_s3g_1 ... done
Stopping ozones3_datanode_1 ... done
Stopping ozones3_scm_1 ... done
Removing ozones3_ozoneManager_1 ... done
Removing ozones3_s3g_1 ... done
Removing ozones3_datanode_1 ... done
Removing ozones3_scm_1 ... done
Removing network ozones3_default
Setting up environment!
Log: /opt/hadoop/smoketest/result/log.html
Report: /opt/hadoop/smoketest/result/report.html
{code}
was (Author: bharatviswa):
Smoke test Output ran locally:
{code:java}
HW13865:smoketest bviswanadham$ robot -v OZONE_S3_SET_CREDENTIALS:false -v
OZONE_TEST:false -v BUCKET:b1-1992 -v DESTBUCKET:b1-1993 -v
ENDPOINT_URL:https://s3.us-east-2.amazonaws.com s3/objectputget.robot
==============================================================================
Objectputget :: S3 gateway test with aws cli
==============================================================================
Put object to s3 | PASS |
------------------------------------------------------------------------------
Get object from s3 ..sh:
md5sum: command not found
.sh: md5sum: command not found
Get object from s3 | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes | PASS |
------------------------------------------------------------------------------
Incorrect values for end and start offset | PASS |
------------------------------------------------------------------------------
Zero byte file | PASS |
------------------------------------------------------------------------------
Objectputget :: S3 gateway test with aws cli | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
Output:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/output.xml
Log:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/log.html
Report:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/report.html
HW13865:smoketest bviswanadham$ ./test.sh --env ozones3 s3/objectputget.robot
-------------------------------------------------
Executing test(s): [s3/objectputget.robot]
Cluster type: ozones3
Compose file:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/../compose/ozones3/docker-compose.yaml
Output dir:
/Users/bviswanadham/workspace/open-hadoop/hadoop/hadoop-ozone/dist/target/ozone-0.4.0-SNAPSHOT/smoketest/result
Command to rerun: ./test.sh --keep --env ozones3 s3/objectputget.robot
-------------------------------------------------
Removing network ozones3_default
WARNING: Network ozones3_default not found.
Creating network "ozones3_default" with the default driver
Creating ozones3_datanode_1 ...
Creating ozones3_scm_1 ...
Creating ozones3_ozoneManager_1 ...
Creating ozones3_s3g_1 ...
Creating ozones3_scm_1
Creating ozones3_datanode_1
Creating ozones3_s3g_1
Creating ozones3_ozoneManager_1 ... done
Waiting 30s for cluster start up...
==============================================================================
Objectputget :: S3 gateway test with aws cli
==============================================================================
Put object to s3 | PASS |
------------------------------------------------------------------------------
Get object from s3 | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(start off... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset(end offse... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with only start offset | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 with both start and endoffset which are... | PASS |
------------------------------------------------------------------------------
Get Partial object from s3 to get last n bytes | PASS |
------------------------------------------------------------------------------
Incorrect values for end and start offset | PASS |
------------------------------------------------------------------------------
Zero byte file | PASS |
------------------------------------------------------------------------------
Objectputget :: S3 gateway test with aws cli | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
Output: /opt/hadoop/smoketest/result/robot-ozones3-s3_objectputget.robot.xml
Stopping ozones3_ozoneManager_1 ... done
Stopping ozones3_s3g_1 ... done
Stopping ozones3_datanode_1 ... done
Stopping ozones3_scm_1 ... done
Removing ozones3_ozoneManager_1 ... done
Removing ozones3_s3g_1 ... done
Removing ozones3_datanode_1 ... done
Removing ozones3_scm_1 ... done
Removing network ozones3_default
Setting up environment!
Log: /opt/hadoop/smoketest/result/log.html
Report: /opt/hadoop/smoketest/result/report.html
{code}
> Support Range header for ozone s3 object download
> -------------------------------------------------
>
> Key: HDDS-791
> URL: https://issues.apache.org/jira/browse/HDDS-791
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: S3
> Reporter: Elek, Marton
> Assignee: Bharat Viswanadham
> Priority: Major
> Attachments: HDDS-791.00.patch, HDDS-791.01.patch
>
>
> Using s3 rest api smaller chunks of an object could be uploaded with using
> Range headers:
> For example:
> {code}
> GET /example-object HTTP/1.1
> Host: example-bucket.s3.amazonaws.com
> x-amz-date: Fri, 28 Jan 2011 21:32:02 GMT
> Range: bytes=0-9
> Authorization: AWS AKIAIOSFODNN7EXAMPLE:Yxg83MZaEgh3OZ3l0rLo5RTX11o=
> Sample Response with Specified Range of the Object Bytes
> {code}
> Can be implemented with using the seek method on OzoneInputStream.
> The Range header support is one of the missing piece for fully support s3a
> interface.
> References:
> Range header spec:
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
> Aws s3 doc:
> https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
