[
https://issues.apache.org/jira/browse/JCLOUDS-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1505.
----------------------------------
Fix Version/s: 2.1.3
2.2.0
Assignee: Andrew Gaul
Resolution: Fixed
> BlobStore.blobMetadata(container, object) returns a StorageMetadata object
> with empty size when using
> org.jclouds.http.apachehc.config.ApacheHCHttpCommandExecutorServiceModule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JCLOUDS-1505
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1505
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-drivers
> Reporter: Xavier BOURGOUIN
> Assignee: Andrew Gaul
> Priority: Major
> Fix For: 2.2.0, 2.1.3
>
> Attachments: jclouds_null_size_reproducer.tar.gz
>
>
> Hello,
>
> When calling *blobStore.blobMetadata(container, objectName).getSize()* with a
> blobStoreContext using ApacheHCHttpCommandExecutorServiceModule, the returned
> BlobMetadata object has a size attribute which is null.
> Issue spotted while using jclouds 2.1.2 over an S3 object store (Scality, but
> seems like it is reproducible with any other S3 provider, and potentially
> others OS providers like GCS, to be confirmed).
> In essence, the minimal reproducer pseudo-code is:
> {code:java}
> BlobStoreContext context = ContextBuilder.newBuilder(provider)
> .credentials(identity, credential)
> .endpoint(endpoint)
> // usage of ApacheHCHttp module
> .modules(ImmutableList.<Module> of(new
> ApacheHCHttpCommandExecutorServiceModule(), new SLF4JLoggingModule()))
> .buildView(BlobStoreContext.class);
>
> try {
> BlobStore blobStore = context.getBlobStore();
> // output is: "File size from metadata: null"
> System.out.println("File size from metadata: " +
> blobStore.blobMetadata(DUMMY_CONTAINER, DUMMY_FILE_NAME).getSize());
> } finally {
> context.close();
> }
> {code}
>
> For convenience, I've attached a ready to run reproducer to this ticket,
> here's how to run it:
>
> {code:bash}
> > mkdir reproducer
> > tar xvzf jclouds_null_size_reproducer.tar.gz -C reproducer
> > cd reproducer
> > mvn install
> # note: pre-req from this stage is to have an S3 Object Store available, you
> can use minio one if you have docker (docker run -it -p 9000:9000 -v
> /mnt/data:/data minio/minio server /data)
> # replace with your Object Store endpoint, identity and access key
> > java -jar target/blobstore-basics-jar-with-dependencies.jar s3
> > http://10.66.33.1:9000 <identity here> <access key here>
> File size from metadata: 8
> File size from metadata (with ApacheHCHttpCommandExecutorServiceModule): null
> {code}
> The reproducer code has jclouds-wire logs enabled (inside log folder) and
> I've also added the log files inside the attached tarball for convenience. We
> see that in the HTTP HEAD response (the 2nd one, which correspond to my
> .blobMetadata() call using ApacheHCHttpCommandExecutorServiceModule), the
> Content-* headers are missing :
> {code:bash}
> >cat log/jclouds-wire.log
> # note: first .blobMetadata() call without
> ApacheHCHttpCommandExecutorServiceModule, the response contains Content-*
> headers
> 2019-07-31 17:30:13,257 DEBUG [jclouds.headers] [main] >> HEAD
> http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
> 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul
> 2019 15:30:13 GMT
> 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Authorization: AWS
> 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Server:
> MinIO/RELEASE.2019-07-24T02-02-23Z
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << ETag:
> "35a41457219c775659b6018fb7f465a1-1"
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Xss-Protection:
> 1; mode=block
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] <<
> Content-Security-Policy: block-all-mixed-content
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Vary: Origin
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Last-Modified: Wed,
> 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul
> 2019 15:30:13 GMT
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id:
> 15B687995977CC49
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Type:
> application/unknown
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Length: 8
> # note: second call with ApacheHCHttpCommandExecutorServiceModule, the
> reponse is missing the content-* headers
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> HEAD
> http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul
> 2019 15:30:13 GMT
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Authorization: AWS
> 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] <<
> Content-Security-Policy: block-all-mixed-content
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << ETag:
> "35a41457219c775659b6018fb7f465a1-1"
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Last-Modified: Wed,
> 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Server:
> MinIO/RELEASE.2019-07-24T02-02-23Z
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Vary: Origin
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id:
> 15B6879961644613
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Xss-Protection:
> 1; mode=block
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul
> 2019 15:30:13 GMT
> {code}
>
> I suspect the issue is coming from
> [ApacheHCHttpCommandExecutorService.java#invoke|https://github.com/apache/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCHttpCommandExecutorService.java#L87]
> method, it seems to be expecting the content-* headers are exclusively
> coming from the reponse.getEntity() (which I believe is null in case of a
> response to a HEAD request), because it strips it out of the headers list
> (which are well containing the content-* headers before the call to
> "filterOutContentHeaders"). A possible fix to cope with all requests kinds
> could be to strip it out *only* if it is already part of the payload (which I
> guess was the initial intent).
>
> Environment details:
> * jclouds version: 2.1.2 (but it is probably reproducible on older versions,
> to be confirmed)
> * Operating system: Linux (kernel 4.15.0-55)
> * Storage provider: S3 (reproduced on both Scality and MinIO), probably
> reproducible with other storage providers involving HTTP HEAD requests to get
> the metadata (GCS, ..)
> * Java version:
> {code:java}
> >java -version{code}
> *
> {code:java}
> java version "1.8.0_201"
> Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
> Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
