[ https://issues.apache.org/jira/browse/JCLOUDS-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Gaul resolved JCLOUDS-1505. ---------------------------------- Fix Version/s: 2.1.3 2.2.0 Assignee: Andrew Gaul Resolution: Fixed > BlobStore.blobMetadata(container, object) returns a StorageMetadata object > with empty size when using > org.jclouds.http.apachehc.config.ApacheHCHttpCommandExecutorServiceModule > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: JCLOUDS-1505 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1505 > Project: jclouds > Issue Type: Bug > Components: jclouds-drivers > Reporter: Xavier BOURGOUIN > Assignee: Andrew Gaul > Priority: Major > Fix For: 2.2.0, 2.1.3 > > Attachments: jclouds_null_size_reproducer.tar.gz > > > Hello, > > When calling *blobStore.blobMetadata(container, objectName).getSize()* with a > blobStoreContext using ApacheHCHttpCommandExecutorServiceModule, the returned > BlobMetadata object has a size attribute which is null. > Issue spotted while using jclouds 2.1.2 over an S3 object store (Scality, but > seems like it is reproducible with any other S3 provider, and potentially > others OS providers like GCS, to be confirmed). > In essence, the minimal reproducer pseudo-code is: > {code:java} > BlobStoreContext context = ContextBuilder.newBuilder(provider) > .credentials(identity, credential) > .endpoint(endpoint) > // usage of ApacheHCHttp module > .modules(ImmutableList.<Module> of(new > ApacheHCHttpCommandExecutorServiceModule(), new SLF4JLoggingModule())) > .buildView(BlobStoreContext.class); > > try { > BlobStore blobStore = context.getBlobStore(); > // output is: "File size from metadata: null" > System.out.println("File size from metadata: " + > blobStore.blobMetadata(DUMMY_CONTAINER, DUMMY_FILE_NAME).getSize()); > } finally { > context.close(); > } > {code} > > For convenience, I've attached a ready to run reproducer to this ticket, > here's how to run it: > > {code:bash} > > mkdir reproducer > > tar xvzf jclouds_null_size_reproducer.tar.gz -C reproducer > > cd reproducer > > mvn install > # note: pre-req from this stage is to have an S3 Object Store available, you > can use minio one if you have docker (docker run -it -p 9000:9000 -v > /mnt/data:/data minio/minio server /data) > # replace with your Object Store endpoint, identity and access key > > java -jar target/blobstore-basics-jar-with-dependencies.jar s3 > > http://10.66.33.1:9000 <identity here> <access key here> > File size from metadata: 8 > File size from metadata (with ApacheHCHttpCommandExecutorServiceModule): null > {code} > The reproducer code has jclouds-wire logs enabled (inside log folder) and > I've also added the log files inside the attached tarball for convenience. We > see that in the HTTP HEAD response (the 2nd one, which correspond to my > .blobMetadata() call using ApacheHCHttpCommandExecutorServiceModule), the > Content-* headers are missing : > {code:bash} > >cat log/jclouds-wire.log > # note: first .blobMetadata() call without > ApacheHCHttpCommandExecutorServiceModule, the response contains Content-* > headers > 2019-07-31 17:30:13,257 DEBUG [jclouds.headers] [main] >> HEAD > http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1 > 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul > 2019 15:30:13 GMT > 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Authorization: AWS > 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE= > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Server: > MinIO/RELEASE.2019-07-24T02-02-23Z > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << ETag: > "35a41457219c775659b6018fb7f465a1-1" > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Xss-Protection: > 1; mode=block > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << > Content-Security-Policy: block-all-mixed-content > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Vary: Origin > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, > 31 Jul 2019 15:30:13 GMT > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul > 2019 15:30:13 GMT > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: > 15B687995977CC49 > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Type: > application/unknown > 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Length: 8 > # note: second call with ApacheHCHttpCommandExecutorServiceModule, the > reponse is missing the content-* headers > 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> HEAD > http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1 > 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul > 2019 15:30:13 GMT > 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Authorization: AWS > 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE= > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << > Content-Security-Policy: block-all-mixed-content > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << ETag: > "35a41457219c775659b6018fb7f465a1-1" > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, > 31 Jul 2019 15:30:13 GMT > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Server: > MinIO/RELEASE.2019-07-24T02-02-23Z > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Vary: Origin > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: > 15B6879961644613 > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Xss-Protection: > 1; mode=block > 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul > 2019 15:30:13 GMT > {code} > > I suspect the issue is coming from > [ApacheHCHttpCommandExecutorService.java#invoke|https://github.com/apache/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCHttpCommandExecutorService.java#L87] > method, it seems to be expecting the content-* headers are exclusively > coming from the reponse.getEntity() (which I believe is null in case of a > response to a HEAD request), because it strips it out of the headers list > (which are well containing the content-* headers before the call to > "filterOutContentHeaders"). A possible fix to cope with all requests kinds > could be to strip it out *only* if it is already part of the payload (which I > guess was the initial intent). > > Environment details: > * jclouds version: 2.1.2 (but it is probably reproducible on older versions, > to be confirmed) > * Operating system: Linux (kernel 4.15.0-55) > * Storage provider: S3 (reproduced on both Scality and MinIO), probably > reproducible with other storage providers involving HTTP HEAD requests to get > the metadata (GCS, ..) > * Java version: > {code:java} > >java -version{code} > * > {code:java} > java version "1.8.0_201" > Java(TM) SE Runtime Environment (build 1.8.0_201-b09) > Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) > {code} > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)