[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=210266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210266
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 08/Mar/19 17:55
Start Date: 08/Mar/19 17:55
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210266)
Time Spent: 4h 20m (was: 4h 10m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=210264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210264
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 08/Mar/19 17:54
Start Date: 08/Mar/19 17:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-471017574
Thank You @elek for the info.
By increasing the docker memory settings, the tests are now passing.
+1.
I will commit this shortly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210264)
Time Spent: 4h 10m (was: 4h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=209497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209497
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 07/Mar/19 11:02
Start Date: 07/Mar/19 11:02
Worklog Time Spent: 10m
Work Description: elek commented on issue #549: HDDS-1213. Support plain
text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470483141
@vivekratnavel thanks to check it
I tried to reproduce the problem I uploaded 100MB file from my host machine
with/without the patch and it worked well.
It's also working from acceptance tests both on my side and on the jenkins.
Can you please share more details? How do you run the tests? What is the os?
What is your awscli/docker version. Do you have any memory limitation on the
virtual host which runs docker?
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209497)
Time Spent: 3h 50m (was: 3h 40m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=209498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209498
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 07/Mar/19 11:03
Start Date: 07/Mar/19 11:03
Worklog Time Spent: 10m
Work Description: elek commented on issue #549: HDDS-1213. Support plain
text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470483470
Can you please try to increase your allocated memory for the containers?
https://docs.docker.com/docker-for-mac/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209498)
Time Spent: 4h (was: 3h 50m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=209286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209286
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 07/Mar/19 01:39
Start Date: 07/Mar/19 01:39
Worklog Time Spent: 10m
Work Description: vivekratnavel commented on issue #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470348018
I am hitting the same error as @bharatviswa504
@elek Please find attached the log from docker-compose
[HDDS-1213.log](https://github.com/apache/hadoop/files/2939068/HDDS-1213.log)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209286)
Time Spent: 3h 40m (was: 3.5h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[ https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=209066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209066 ] ASF GitHub Bot logged work on HDDS-1213: Author: ASF GitHub Bot Created on: 06/Mar/19 19:10 Start Date: 06/Mar/19 19:10 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #549: HDDS-1213. Support plain text S3 MPU initialization request URL: https://github.com/apache/hadoop/pull/549#issuecomment-470236321 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 26 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1026 | trunk passed | | -1 | compile | 36 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 30 | trunk passed | | +1 | mvnsite | 56 | trunk passed | | +1 | shadedclient | 733 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 40 | trunk passed | | +1 | javadoc | 39 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 11 | Maven dependency ordering for patch | | -1 | mvninstall | 18 | dist in the patch failed. | | -1 | compile | 26 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 18 | the patch passed | | +1 | mvnsite | 46 | the patch passed | | -1 | whitespace | 0 | The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 | whitespace | 1 | The patch 19849 line(s) with tabs. | | +1 | shadedclient | 1000 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 46 | the patch passed | | +1 | javadoc | 40 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 37 | s3gateway in the patch passed. | | +1 | unit | 21 | dist in the patch passed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 3430 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/549 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f1b54cd51a83 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2c3ec37 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/branch-compile-hadoop-ozone.txt | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/patch-compile-hadoop-ozone.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/artifact/out/whitespace-tabs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/4/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=209020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-209020
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 06/Mar/19 18:20
Start Date: 06/Mar/19 18:20
Worklog Time Spent: 10m
Work Description: elek commented on issue #549: HDDS-1213. Support plain
text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470217716
bq. But platform change is related to this patch, as previously we used to
see check platform type and run random create file.
Oh, I got it finally. You are right. I changed it to use raw bytes instead
of M or m postfixes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 209020)
Time Spent: 3h 20m (was: 3h 10m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208993
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 06/Mar/19 17:25
Start Date: 06/Mar/19 17:25
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470198311
@elek
But any idea how to resolve the above mentioned error which is happening on
my system when i run my tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208993)
Time Spent: 3h 10m (was: 3h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208992
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 06/Mar/19 17:24
Start Date: 06/Mar/19 17:24
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470197883
Hi @elek
Thanks for the update.
> White space problems are reported by the yetus. I committed to a fix in a
separated commit (a086953) to make it easier to review.
I am fine with it.
But platform change is related to this patch, as previosuly we used to see
check platform type and run random create file. As now you have changed it, so
we don't need the check. So, that is the reason for the comments.
+1 LGTM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208992)
Time Spent: 3h (was: 2h 50m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208968
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 06/Mar/19 16:55
Start Date: 06/Mar/19 16:55
Worklog Time Spent: 10m
Work Description: elek commented on issue #549: HDDS-1213. Support plain
text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-470186740
White space problems are reported by the yetus. I committed to a fix in a
separated commit
(https://github.com/apache/hadoop/pull/549/commits/a086953243b71796bf06022a0775aebcfd29ea52)
to make it easier to review.
I removed the platform lines
(https://github.com/apache/hadoop/pull/549/commits/9f59f5d06960634fcca77b7dd5c07da70dce21ff),
but to be honest, they are also independent from the patch.
So we should either accept both whitespace/platform fixes or create two new
jiras for both. Just to be consistent ;-)
(As they are both committed in two separated commits they can be
reverted/accepted during the merge.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208968)
Time Spent: 2h 50m (was: 2h 40m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208958
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 06/Mar/19 16:48
Start Date: 06/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: elek commented on pull request #549: HDDS-1213. Support
plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r263033449
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -74,17 +70,16 @@ Test Multipart Upload Complete
Should contain ${result}UploadId
#upload parts
- ${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey1 --part-number 1 --body /tmp/part1 --upload-id
${uploadID}
- ${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${system} = Evaluateplatform.system()platform
+Run Keyword Create Random file 5
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey1 --part-number 1 --body /tmp/part1 --upload-id ${uploadID}
+${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Execute echo "Part2" > /tmp/part2
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey1 --part-number 2 --body /tmp/part2 --upload-id
${uploadID}
- ${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey1 --part-number 2 --body /tmp/part2 --upload-id ${uploadID}
+${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Review comment:
Yetus asked me to not use tab. Without fixing the tabs the patch would cause
a whitespace mismatch (my new lines with spaces and the old lines with tabs).
So it's intentional, but I can move to a different
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208958)
Time Spent: 2h 40m (was: 2.5h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[ https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208140 ] ASF GitHub Bot logged work on HDDS-1213: Author: ASF GitHub Bot Created on: 05/Mar/19 22:48 Start Date: 05/Mar/19 22:48 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #549: HDDS-1213. Support plain text S3 MPU initialization request URL: https://github.com/apache/hadoop/pull/549#issuecomment-469889963 Over all patch LGTM. Thank You @elek for offline discussion. Yes, I see that when file size is large, cp is using multipart upload request. Few minor comments: 1. Remove unncessary space indentation changes 2. Removed platform.system(), as we don't need it now. I see below error when running test manually with docker ozones3 cluster on my dev machine. ``` s3g_1 | 2019-03-05 22:37:40 WARN HttpChannel:499 - //localhost:9878/b12345/mm11?partNumber=3=d51657f3-3276-49d0-977a-af9f4bab94bc-19533321172195 s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space s3g_1 |at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) s3g_1 |at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 |at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 |at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 |at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 |at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 |at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 |at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 |at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 |at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 |at java.lang.Thread.run(Thread.java:748) s3g_1 | Caused by: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space s3g_1 |at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) s3g_1 |at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 |at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) s3g_1 |at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) s3g_1 |at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) s3g_1 |at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 |at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 |at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 |at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) s3g_1 |at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 |at
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208138
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 05/Mar/19 22:47
Start Date: 05/Mar/19 22:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262712309
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -74,17 +70,16 @@ Test Multipart Upload Complete
Should contain ${result}UploadId
#upload parts
- ${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey1 --part-number 1 --body /tmp/part1 --upload-id
${uploadID}
- ${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${system} = Evaluateplatform.system()platform
+Run Keyword Create Random file 5
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey1 --part-number 1 --body /tmp/part1 --upload-id ${uploadID}
+${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Execute echo "Part2" > /tmp/part2
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey1 --part-number 2 --body /tmp/part2 --upload-id
${uploadID}
- ${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey1 --part-number 2 --body /tmp/part2 --upload-id ${uploadID}
+${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Review comment:
Are these unintended changes? As I see the change is indentation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208138)
Time Spent: 2h 10m (was: 2h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
>
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[ https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208134 ] ASF GitHub Bot logged work on HDDS-1213: Author: ASF GitHub Bot Created on: 05/Mar/19 22:47 Start Date: 05/Mar/19 22:47 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #549: HDDS-1213. Support plain text S3 MPU initialization request URL: https://github.com/apache/hadoop/pull/549#issuecomment-469889963 Over all patch LGTM. Thank You @elek for offline discussion. Yes, I see that when file size is large, cp is using multipart upload request. Few minor comments: 1. Remove unncessary space indentation changes 2. Removed platform.system(), as we don't need it now. I see below error when running test manually with docker ozones3 cluster ``` s3g_1 | 2019-03-05 22:37:40 WARN HttpChannel:499 - //localhost:9878/b12345/mm11?partNumber=3=d51657f3-3276-49d0-977a-af9f4bab94bc-19533321172195 s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space s3g_1 |at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) s3g_1 |at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 |at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 |at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 |at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 |at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 |at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 |at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 |at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 |at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 |at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 |at java.lang.Thread.run(Thread.java:748) s3g_1 | Caused by: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space s3g_1 |at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) s3g_1 |at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 |at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 |at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) s3g_1 |at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) s3g_1 |at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) s3g_1 |at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 |at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 |at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 |at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 |at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) s3g_1 |at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 |at
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208139
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 05/Mar/19 22:47
Start Date: 05/Mar/19 22:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262712573
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -109,14 +101,14 @@ Test Multipart Upload Complete Entity too small
#upload parts
Execute echo "Part1" > /tmp/part1
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey2 --part-number 1 --body /tmp/part1 --upload-id
${uploadID}
- ${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey2 --part-number 1 --body /tmp/part1 --upload-id ${uploadID}
+${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Execute echo "Part2" > /tmp/part2
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey2 --part-number 2 --body /tmp/part2 --upload-id
${uploadID}
- ${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${result} = Execute AWSS3APICli upload-part --bucket ${BUCKET}
--key multipartKey2 --part-number 2 --body /tmp/part2 --upload-id ${uploadID}
+${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
+Should contain ${result}ETag
Review comment:
Can we remove these indentation changes?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208139)
Time Spent: 2h 20m (was: 2h 10m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208137
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 05/Mar/19 22:47
Start Date: 05/Mar/19 22:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262712695
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -171,37 +163,43 @@ Test list parts
Should contain ${result}UploadId
#upload parts
- ${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey5 --part-number 1 --body /tmp/part1 --upload-id
${uploadID}
- ${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
-
- Execute echo "Part2" > /tmp/part2
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey5 --part-number 2 --body /tmp/part2 --upload-id
${uploadID}
- ${eTag2} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${system} = Evaluateplatform.system()platform
Review comment:
Remove this line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208137)
Time Spent: 2h 10m (was: 2h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208135
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 05/Mar/19 22:47
Start Date: 05/Mar/19 22:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262711494
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -54,16 +52,14 @@ Test Multipart Upload
# upload we get error entity too small. So, considering further complete
# multipart upload, uploading each part as 5MB file, exception is for last part
- ${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey --part-number 1 --body /tmp/part1 --upload-id
${nextUploadID}
- Should contain ${result}ETag
+${system} = Evaluateplatform.system()platform
Review comment:
We can remove line 55, as we have previously checked system type and created
random file.
Now it is removed, we don't need it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208135)
Time Spent: 1h 50m (was: 1h 40m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=208136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-208136
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 05/Mar/19 22:47
Start Date: 05/Mar/19 22:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #549:
HDDS-1213. Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262711715
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -74,17 +70,16 @@ Test Multipart Upload Complete
Should contain ${result}UploadId
#upload parts
- ${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
- ${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey1 --part-number 1 --body /tmp/part1 --upload-id
${uploadID}
- ${eTag1} = Execute and checkrc echo '${result}' | jq -r
'.ETag' 0
- Should contain ${result}ETag
+${system} = Evaluateplatform.system()platform
Review comment:
We can remove this line. Same comment as above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 208136)
Time Spent: 2h (was: 1h 50m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[ https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207740 ] ASF GitHub Bot logged work on HDDS-1213: Author: ASF GitHub Bot Created on: 05/Mar/19 10:35 Start Date: 05/Mar/19 10:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #549: HDDS-1213. Support plain text S3 MPU initialization request URL: https://github.com/apache/hadoop/pull/549#issuecomment-469629985 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 19 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 47 | Maven dependency ordering for branch | | +1 | mvninstall | 1064 | trunk passed | | -1 | compile | 45 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 26 | trunk passed | | -1 | mvnsite | 29 | dist in trunk failed. | | +1 | shadedclient | 815 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 34 | trunk passed | | +1 | javadoc | 37 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 12 | Maven dependency ordering for patch | | -1 | mvninstall | 17 | dist in the patch failed. | | -1 | compile | 41 | hadoop-ozone in the patch failed. | | -1 | javac | 41 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 19 | the patch passed | | -1 | mvnsite | 19 | dist in the patch failed. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 808 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 39 | the patch passed | | +1 | javadoc | 33 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 33 | s3gateway in the patch passed. | | -1 | unit | 22 | dist in the patch failed. | | +1 | asflicense | 25 | The patch does not generate ASF License warnings. | | | | 3329 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/549 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1c6f25d86f31 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0aefe28 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/branch-compile-hadoop-ozone.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/branch-mvnsite-hadoop-ozone_dist.txt | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/patch-compile-hadoop-ozone.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/patch-mvnsite-hadoop-ozone_dist.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/artifact/out/patch-unit-hadoop-ozone_dist.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/testReport/ | | Max. process+thread count | 327 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/3/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207275
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 16:48
Start Date: 04/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262147210
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -55,13 +53,11 @@ Test Multipart Upload
# multipart upload, uploading each part as 5MB file, exception is for last part
${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
+ Run Keyword Create Random file 5
Review comment:
whitespace:tabs in line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207275)
Time Spent: 50m (was: 40m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207276
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 16:48
Start Date: 04/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262147219
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -55,13 +53,11 @@ Test Multipart Upload
# multipart upload, uploading each part as 5MB file, exception is for last part
${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
+ Run Keyword Create Random file 5
${result} = Execute AWSS3APICli upload-part --bucket
${BUCKET} --key multipartKey --part-number 1 --body /tmp/part1 --upload-id
${nextUploadID}
Should contain ${result}ETag
# override part
- Run Keyword if '${system}' == 'Darwin'Create Random file for
mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
+ Run Keyword Create Random file 5
Review comment:
whitespace:tabs in line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207276)
Time Spent: 1h (was: 50m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207274
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 16:48
Start Date: 04/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262147199
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -205,3 +196,10 @@ Test list parts
#finally abort it
${result} = Execute AWSS3APICli and checkrc
abort-multipart-upload --bucket ${BUCKET} --key multipartKey5 --upload-id
${uploadID}0
+
+Test Multipart Upload with the simplified aws s3 cp API
+ Create Random file 22
Review comment:
whitespace:tabs in line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207274)
Time Spent: 40m (was: 0.5h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[ https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207278=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207278 ] ASF GitHub Bot logged work on HDDS-1213: Author: ASF GitHub Bot Created on: 04/Mar/19 16:48 Start Date: 04/Mar/19 16:48 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #549: HDDS-1213. Support plain text S3 MPU initialization request URL: https://github.com/apache/hadoop/pull/549#issuecomment-469325868 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 67 | Maven dependency ordering for branch | | +1 | mvninstall | 1206 | trunk passed | | -1 | compile | 117 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 28 | trunk passed | | -1 | mvnsite | 30 | dist in trunk failed. | | +1 | shadedclient | 870 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 41 | trunk passed | | +1 | javadoc | 39 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 13 | Maven dependency ordering for patch | | -1 | mvninstall | 20 | dist in the patch failed. | | -1 | compile | 100 | hadoop-ozone in the patch failed. | | -1 | javac | 100 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 24 | the patch passed | | -1 | mvnsite | 22 | dist in the patch failed. | | -1 | whitespace | 0 | The patch 5 line(s) with tabs. | | +1 | shadedclient | 860 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/dist | | +1 | findbugs | 51 | the patch passed | | +1 | javadoc | 39 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 40 | s3gateway in the patch passed. | | -1 | unit | 22 | dist in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 3808 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/549 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a01c85c47fa9 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 15098df | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/branch-compile-hadoop-ozone.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/branch-mvnsite-hadoop-ozone_dist.txt | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/patch-compile-hadoop-ozone.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/patch-mvnsite-hadoop-ozone_dist.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/whitespace-tabs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/artifact/out/patch-unit-hadoop-ozone_dist.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/testReport/ | | Max. process+thread count | 306 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-549/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message,
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207273
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 16:48
Start Date: 04/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262147191
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -172,8 +164,7 @@ Test list parts
#upload parts
${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
+ Run Keyword Create Random file 5
Review comment:
whitespace:tabs in line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207273)
Time Spent: 0.5h (was: 20m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207277
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 16:48
Start Date: 04/Mar/19 16:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#discussion_r262147227
##
File path: hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
##
@@ -75,8 +71,7 @@ Test Multipart Upload Complete
#upload parts
${system} = Evaluateplatform.system()platform
- Run Keyword if '${system}' == 'Darwin' Create Random file for mac
- Run Keyword if '${system}' == 'Linux' Create Random file for
linux
+ Run Keyword Create Random file 5
Review comment:
whitespace:tabs in line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207277)
Time Spent: 1h 10m (was: 1h)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207156
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 14:08
Start Date: 04/Mar/19 14:08
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #549: HDDS-1213.
Support plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549#issuecomment-469265127
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 0 | Docker mode activated. |
| -1 | patch | 6 | https://github.com/apache/hadoop/pull/549 does not apply
to trunk. Rebase required? Wrong Branch? See
https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/549 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-549/1/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207156)
Time Spent: 20m (was: 10m)
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1213) Support plain text S3 MPU initialization request
[
https://issues.apache.org/jira/browse/HDDS-1213?focusedWorklogId=207155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-207155
]
ASF GitHub Bot logged work on HDDS-1213:
Author: ASF GitHub Bot
Created on: 04/Mar/19 14:05
Start Date: 04/Mar/19 14:05
Worklog Time Spent: 10m
Work Description: elek commented on pull request #549: HDDS-1213. Support
plain text S3 MPU initialization request
URL: https://github.com/apache/hadoop/pull/549
S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
We have extensive testing with using 'aws s3api' application which is passed.
But it turned out that the more simple `aws s3 cp` command fails with _405
Media type not supported error_ message
The root cause of this issue is the JAXRS implementation of the multipart
upload method:
{code}
@POST
@Produces(MediaType.APPLICATION_XML)
public Response multipartUpload(
@PathParam("bucket") String bucket,
@PathParam("path") String key,
@QueryParam("uploads") String uploads,
@QueryParam("uploadId") @DefaultValue("") String uploadID,
CompleteMultipartUploadRequest request) throws IOException,
OS3Exception {
if (!uploadID.equals("")) {
//Complete Multipart upload request.
return completeMultipartUpload(bucket, key, uploadID, request);
} else {
// Initiate Multipart upload request.
return initiateMultipartUpload(bucket, key);
}
}
{code}
Here we have a CompleteMultipartUploadRequest parameter which is created by
the JAXRS framework based on the media type and the request body. With
_Content-Type: application/xml_ it's easy: the JAXRS framework uses the
built-in JAXB serialization. But with plain/text content-type it's not possible
as there is no serialization support for CompleteMultipartUploadRequest from
plain/text.
See: https://issues.apache.org/jira/browse/HDDS-1213
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 207155)
Time Spent: 10m
Remaining Estimate: 0h
> Support plain text S3 MPU initialization request
>
>
> Key: HDDS-1213
> URL: https://issues.apache.org/jira/browse/HDDS-1213
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> S3 Multi-Part-Upload (MPU) is implemented recently in the Ozone s3 gateway.
> We have extensive testing with using 'aws s3api' application which is passed.
> But it turned out that the more simple `aws s3 cp` command fails with _405
> Media type not supported error_ message
> The root cause of this issue is the JAXRS implementation of the multipart
> upload method:
> {code}
> @POST
> @Produces(MediaType.APPLICATION_XML)
> public Response multipartUpload(
> @PathParam("bucket") String bucket,
> @PathParam("path") String key,
> @QueryParam("uploads") String uploads,
> @QueryParam("uploadId") @DefaultValue("") String uploadID,
> CompleteMultipartUploadRequest request) throws IOException,
> OS3Exception {
> if (!uploadID.equals("")) {
> //Complete Multipart upload request.
> return completeMultipartUpload(bucket, key, uploadID, request);
> } else {
> // Initiate Multipart upload request.
> return initiateMultipartUpload(bucket, key);
> }
> }
> {code}
> Here we have a CompleteMultipartUploadRequest parameter which is created by
> the JAXRS framework based on the media type and the request body. With
> _Content-Type: application/xml_ it's easy: the JAXRS framework uses the
> built-in JAXB serialization. But with plain/text content-type it's not
> possible as there is no serialization support for
> CompleteMultipartUploadRequest from plain/text.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
