Re: ApacheHC driver and stream closing
Hi Ignasi.
In fact, in the patch I provided for the other issue (S3 upload
performance), I already
removed the hashing lines and simply took the provided MD5 hash from the
ContentMetadata.
And that worked. But I wasn't sure about the intention of the additional
hashing,
so I thought I better ask for more insight here ;).
So no changes were needed for [1] and [2]. If it makes sense, I can
create a PR so someone
can review the changes.
Thanks,
Veit
Am 11.03.2015 um 22:23 schrieb Ignasi Barrera:
Looking at the code I would say it is an issue, but at some point that
should be intentional. The Apache HC driver explicitly excludes the
Content-MD5 header when converting the jclouds request to its request
object (see [1]).
By looking at the other drivers (the default Java one and OkHttp),
both use that method to populate the Content-MD5 header (here you can
see what it does [2]) which does pretty much the same than what the
ApacheHC does *after*, by computing the hash again.
Could you try removing the lines that rehash the stream and populate
the Content-MD5 header from the Apache HC driver, and add the
Content-MD5 header to the list of allowed headers in the first link?
I'm pretty sure that would work. And if that works, mind opening a
pull request with the fix?
HTH!
I.
[1]
https://github.com/jclouds/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCUtils.java#L155-L162
[2]
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/io/ContentMetadataCodec.java#L98-L99
On 11 March 2015 at 17:22, Veit Guna veit.g...@gmx.de wrote:
Hi.
I have a problem with the ApacheHC driver using jclouds 1.8.1. I'm using it
to upload files
to an S3 bucket. For the payload I'm using an InputStream that is not
resettable nor rereadable.
When I try to send the file via:
PayloadBlobBuilder blobBuilder = blobStore.blobBuilder(test)
.payload(inputStream)
.contentDisposition(attachment; filename=test)
.contentType(application/octet-stream)
.contentMD5(hash)
.contentLength(file.length()
);
It dies with the exception below.
A quick look at ApacheHCHttpCommandExecutorService shows:
@Override
protected HttpUriRequest convert(HttpRequest request) throws IOException {
HttpUriRequest returnVal =
apacheHCUtils.convertToApacheRequest(request);
if (request.getPayload() != null
request.getPayload().getContentMetadata().getContentMD5() != null) {
String md5 =
base64().encode(ByteStreams2.hashAndClose(request.getPayload().openStream(),
md5()).asBytes());
returnVal.addHeader(Content-MD5, md5);
}
return returnVal;
}
That means, that if an MD5 is specified, it reads the whole(!) stream to
create his own MD5. Why is that? The client already
supplied an MD5. So why hashing it again?
Does that mean, using Md5 together with ApacheHC and non rereadable
streams isn't working at all? Is that by design?
The normal JavaUrlHttp driver works like a charm though.
Maybe someone could shed some light on this :)?
Thanks!
Veit
Exception in thread main org.jclouds.http.HttpResponseException: null
connecting to PUT https://mybucket.s3-eu-west-1.amazonaws.com/test HTTP/1.1
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:110)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.putObject(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:235)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:95)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:213)
at org.jclouds.test.S3UploadTest.performUpload(S3UploadTest.java:71)
at org.jclouds.test.S3UploadTest.main(S3UploadTest.java:36)
Caused by: org.apache.http.client.ClientProtocolException
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at
org.jclouds.http.apachehc.ApacheHCHttpCommandExecutorService.executeRequest(ApacheHCHttpCommandExecutorService.java:109)
at
Re: ApacheHC driver and stream closing
A PR would be much appreciated :) It is also easier to discuss
impl/design things there.
Thanks!
On 11 March 2015 at 22:47, Veit Guna veit.g...@gmx.de wrote:
Hi Ignasi.
In fact, in the patch I provided for the other issue (S3 upload
performance), I already
removed the hashing lines and simply took the provided MD5 hash from the
ContentMetadata.
And that worked. But I wasn't sure about the intention of the additional
hashing,
so I thought I better ask for more insight here ;).
So no changes were needed for [1] and [2]. If it makes sense, I can
create a PR so someone
can review the changes.
Thanks,
Veit
Am 11.03.2015 um 22:23 schrieb Ignasi Barrera:
Looking at the code I would say it is an issue, but at some point that
should be intentional. The Apache HC driver explicitly excludes the
Content-MD5 header when converting the jclouds request to its request
object (see [1]).
By looking at the other drivers (the default Java one and OkHttp),
both use that method to populate the Content-MD5 header (here you can
see what it does [2]) which does pretty much the same than what the
ApacheHC does *after*, by computing the hash again.
Could you try removing the lines that rehash the stream and populate
the Content-MD5 header from the Apache HC driver, and add the
Content-MD5 header to the list of allowed headers in the first link?
I'm pretty sure that would work. And if that works, mind opening a
pull request with the fix?
HTH!
I.
[1]
https://github.com/jclouds/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCUtils.java#L155-L162
[2]
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/io/ContentMetadataCodec.java#L98-L99
On 11 March 2015 at 17:22, Veit Guna veit.g...@gmx.de wrote:
Hi.
I have a problem with the ApacheHC driver using jclouds 1.8.1. I'm using it
to upload files
to an S3 bucket. For the payload I'm using an InputStream that is not
resettable nor rereadable.
When I try to send the file via:
PayloadBlobBuilder blobBuilder = blobStore.blobBuilder(test)
.payload(inputStream)
.contentDisposition(attachment; filename=test)
.contentType(application/octet-stream)
.contentMD5(hash)
.contentLength(file.length()
);
It dies with the exception below.
A quick look at ApacheHCHttpCommandExecutorService shows:
@Override
protected HttpUriRequest convert(HttpRequest request) throws IOException
{
HttpUriRequest returnVal =
apacheHCUtils.convertToApacheRequest(request);
if (request.getPayload() != null
request.getPayload().getContentMetadata().getContentMD5() != null) {
String md5 =
base64().encode(ByteStreams2.hashAndClose(request.getPayload().openStream(),
md5()).asBytes());
returnVal.addHeader(Content-MD5, md5);
}
return returnVal;
}
That means, that if an MD5 is specified, it reads the whole(!) stream to
create his own MD5. Why is that? The client already
supplied an MD5. So why hashing it again?
Does that mean, using Md5 together with ApacheHC and non rereadable
streams isn't working at all? Is that by design?
The normal JavaUrlHttp driver works like a charm though.
Maybe someone could shed some light on this :)?
Thanks!
Veit
Exception in thread main org.jclouds.http.HttpResponseException: null
connecting to PUT https://mybucket.s3-eu-west-1.amazonaws.com/test HTTP/1.1
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:110)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.putObject(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:235)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:95)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:213)
at org.jclouds.test.S3UploadTest.performUpload(S3UploadTest.java:71)
at org.jclouds.test.S3UploadTest.main(S3UploadTest.java:36)
Caused by: org.apache.http.client.ClientProtocolException
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at
