[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120453#comment-17120453
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 6e6f8ebf779d8edc5cedec687558637d8212ab18 in jclouds's branch
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=6e6f8eb ]
JCLOUDS-912: JCLOUDS-1547: GCS InputStream single-part upload
Previously this provider worked around a RestAnnotationProcessor quirk
by using multi-part uploads for InputStream payloads. Instead work
around the quirk another way which allows a single-part upload. This
allows inclusion of the Content-MD5 header during object creation.
Backfill tests with both ByteSource and InputStream inputs.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Time Spent: 20m
> Remaining Estimate: 0h
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359738#comment-16359738
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 88c84af8788116be2b1f492c3314e07b493dc6f2 in jclouds's branch
refs/heads/master from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=88c84af ]
JCLOUDS-912: Use simpleUpload in GCS BlobStore MPU
This works around a regression with InputStream payloads that I cannot
track down. Using simpleUpload instead of multipartUpload is more
optimal since the former avoids creating a multipart form request.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
>Priority: Major
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115931#comment-16115931
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 69866189d73449aeb5b698e30959b5fcfaeab0cc in jclouds-labs-google's branch
refs/heads/2.0.x from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds-labs-google.git;h=6986618 ]
JCLOUDS-1327: Do not try GCS MPU if length is zero
References JCLOUDS-912.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115927#comment-16115927
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 9e73bbec16cbf91c5f228d4c3ae99c6c526081f8 in jclouds's branch
refs/heads/master from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=9e73bbe ]
JCLOUDS-1327: Do not try GCS MPU if length is zero
References JCLOUDS-912.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976202#comment-15976202
]
Andrew Gaul commented on JCLOUDS-912:
-
We have a workaround so this should not block graduation to core.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
>Assignee: Daniel Broudy
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314667#comment-15314667
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 0673713c6dd0d21fedf25497c507b521c32996e5 in jclouds-labs-google's branch
refs/heads/master from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds-labs-google.git;h=0673713 ]
JCLOUDS-912: GCS upload InputStream work around
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
>Assignee: Daniel Broudy
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313604#comment-15313604
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit f7596fbc219cd3cf32cba9cac31524ad9ca62866 in jclouds-labs-google's branch
refs/heads/master from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds-labs-google.git;h=f7596fb ]
JCLOUDS-912: Remove GCS simpleUpload UploadBinder
This is not necessary; jclouds binds the Payload automatically;
jclouds binds the Payload automatically.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
>Assignee: Daniel Broudy
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atla
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569519#comment-14569519
]
ASF subversion and git services commented on JCLOUDS-912:
-
Commit 1a64a1f0fe2895cba5af810b4747438737439fcd in jclouds's branch
refs/heads/1.9.x from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=1a64a1f ]
JCLOUDS-912: Implement RandomInputStream.close
Prevent reading closed RandomInputStream.
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560174#comment-14560174
]
Andrew Gaul commented on JCLOUDS-912:
-
[~Bhash90] What do you think about [~bpiper] 's suggestions?
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560161#comment-14560161
]
Ben Piper commented on JCLOUDS-912:
---
For simple uploads, I would suggest that UploadBinder is completely redundant.
The multipart upload is a bit trickier though, since RestAnnotationProcessor
can't construct the MultipartForm payload itself, and thus
MultipartUploadBinder has to step in - which it unfortunately can't do without
replacing the payload that RAP created and thus closing any underlying
InputStream.
* Given that RAP already handles PartParam annotations, it seems preferable to
work with this - the only issue being that currently an ObjectTemplate is
provided instead of a StringPayload, which is a convenience, though a helper
method that does the conversion would also be convenient - nevertheless
changing the method signature may be undesirable from a backwards compatibility
perspective. Alternatively, perhaps a similar approach to PathParam and
ParamParser could be taken with PartParam - e.g. in
RestAnnotationParser.getParts, support a PayloadParser annotation that
specifies a Function which turns an argument into a Payload.
* Depending on how necessary it is to release the previously set payload,
either PayloadEnclosingImpl could not call release(), or it could provide
another method that doesn't call it. This could alternatively be done in
HttpRequest or HttpMessage - in either case I wouldn't have thought that we
would be using multiple payloads over the lifecycle of the object, or indeed
would care about whether they are closed.
I think the key architectural decision here is whether this is about MapBinders
not being able to mutate the payload non-destructively, or about RAP.apply not
being able to build up the MultipartForm payload itself and achieve a
satisfactory result in the case of the GCS API. Either way, it may be
difficult to avoid fixing this nicely without a change to jclouds-core.
Perhaps [~gaul] and/or [~broudy] could weigh in on the above. For me
personally, getting rid of UploadBinder allows simpleUpload to work with
streams (so offers a workaround of sorts for multipart uploads), and even if
the multipart solution takes more time, that may be a good place to start
(unless someone knows of a scenario in which UploadBinder isn't redundant).
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.re
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558717#comment-14558717
]
Andrew Gaul commented on JCLOUDS-912:
-
Great diagnosis [~bpiper]! I corrected the {{RandomInputStream}}
implementation and now
{{GoogleCloudStorageBlobIntegrationLiveTest.testPutInputStream}} fails as you
predicted. Can you investigate this issue further and send us a pull request
with a fix? If you need any help please message gaul or broudy on
irc.freenode.net #jclouds. Thanks!
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558687#comment-14558687
]
Ben Piper commented on JCLOUDS-912:
---
Probably because they use a TestUtils.RandomInputStream, for which close() does
nothing. It may be worth having one or two test cases that use input streams
which cannot be read from after a close().
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-912) GCS uploads with InputStream payloads are not working
[
https://issues.apache.org/jira/browse/JCLOUDS-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558683#comment-14558683
]
Andrew Gaul commented on JCLOUDS-912:
-
{{BaseBlobIntegrationTest.testPutMultipartInputStream}} should exercise this
functionality -- I wonder why it does not show these symptoms?
> GCS uploads with InputStream payloads are not working
> -
>
> Key: JCLOUDS-912
> URL: https://issues.apache.org/jira/browse/JCLOUDS-912
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-labs-google
>Affects Versions: 1.9.0
> Environment: Ubuntu 14.04.2 LTS 64-bit
> java version "1.7.0_76"
> Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
>Reporter: Ben Piper
> Labels: google-cloud-storage
>
> It seems that Payloads based on an InputStream are not currently working with
> the google-cloud-storage API, either with simpleUpload or multipartUpload.
> The stream is getting closed before a single byte is read for transmission to
> the GCS endpoint.
> The payload in the request object is first initialised when
> GeneratedHttpRequest.Builder.build is called in
> RestAnnotationProcessor.apply, and then setPayload is later called on the
> HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of
> simpleUpload), causing the originally supplied InputStream to be closed.
> The setPayload call in UploadBinder.bindToRequest seems redundant, if the
> payload has already been set when the request object was built.
> MultipartUploadBinder.bindToRequest wraps the original payload in a
> MultipartForm, but the stream in the media part will still end up being
> closed by the setPayload call. At face value, it doesn't seem like
> MapBinder.bindToRequest is an appropriate place to call setPayload unless it
> is legitimate to completely replace the original payload (rather than wrap
> it) - or at least if it has to mutate the payload property of the request
> object, then it needs some way of doing it that doesn't result in the
> originally supplied Payload being 'released'.
> The simplest way of reproducing this is probably just to add a new
> integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but
> using a FileInputStream based Payload rather than a ByteSource one.
> Relevant portion of stack trace is below.
> {code}
> Caused by: java.io.IOException: Stream already closed
> at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248)
> ~[mimepull-1.9.3.jar:1.9.3]
> at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219)
> ~[mimepull-1.9.3.jar:1.9.3]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:208)
> ~[na:1.7.0_76]
> at java.io.SequenceInputStream.read(SequenceInputStream.java:211)
> ~[na:1.7.0_76]
> at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
> ~[guava-16.0.1.jar:na]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> ~[jclouds-core-1.9.0.jar:1.9.0]
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> ~[guava-16.0.1.jar:na]
> at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
