[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296462#comment-17296462 ] Andrew Gaul commented on JCLOUDS-1504: -- [~theghost5800] would you like to try fixing this? > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230627#comment-17230627 ] Kristian Atanasov commented on JCLOUDS-1504: I did create some test application which run in separate thread create/upload blob in s3-blobstore, in main thread execute list method including metadata and wait until thread finish work, after that again in separate thread execute removeBlob method and in main thread execute list method including metadata and then fails with NPE due to returned null for blob metadata. So I don't see what further investigation is necessary, a fix should be applied in jclouds library and return set of blobs with or empty metadata map. > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229958#comment-17229958 ] Kristian Atanasov commented on JCLOUDS-1504: Hello [~gaul], I hit the same issue with my application using aws-s3 objectstore with jclouds. It does not occur on every {code:java} PageSet list(String container, ListContainerOptions options);{code} method call but when it occurs, throws the following exception: {code:java}java.lang.NullPointerException: Cannot invoke "org.jclouds.blobstore.domain.StorageMetadata.getName()" because "storageMetadata" is null at org.jclouds.blobstore.strategy.internal.FetchBlobMetadata.apply(FetchBlobMetadata.java:112) at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:178) at org.cloudfoundry.multiapps.controller.persistence.services.ObjectStoreFileStorage.getEntryNames(ObjectStoreFileStorage.java:188) at org.cloudfoundry.multiapps.controller.persistence.services.ObjectStoreFileStorage.removeBlobsByFilter(ObjectStoreFileStorage.java:180) at org.cloudfoundry.multiapps.controller.persistence.services.ObjectStoreFileStorage.deleteFilesBySpace(ObjectStoreFileStorage.java:90) at org.cloudfoundry.multiapps.controller.persistence.services.FileService.deleteBySpace(FileService.java:133){code} I am using jclouds version 2.2.1 . I have tried to filter nulls from PageSet but this won't help because in latest version there is [new change|https://github.com/apache/jclouds/commit/5ac92111c4e9347c492a16ecd402acafdb1f9cd7#diff-e5fdbb44b36c6fd2f70b7cb31c38a6fc6facb0d2c8141a3e558538b7d6aecbbbR112] which is related with [list() result order|https://issues.apache.org/jira/browse/JCLOUDS-1543]. Now I can't filter out null elements from PageSet in my code because the issue is inside jclouds library. So what I can do to avoid this NPE and retrieve blobs with metadata together without any additional performance impact? Thanks, Kristian > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907443#comment-16907443 ] Andrew Gaul commented on JCLOUDS-1504: -- You can find an explanation of eventual consistency here: https://github.com/gaul/are-we-consistent-yet I am sorry you feel like your questions have not been answered, although I have suggested an alternate and more correct fix. I don't have the time to investigate this myself and no other user has reported these symptoms so you you will need to do this work yourself. I recommend re-reading my responses which have several suggestions. > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906885#comment-16906885 ] Енчо Белезирев commented on JCLOUDS-1504: - Hi [~gaul], I have explained above, the nulls are produced for the 404 Not Found HTTP Statuses returned when querying the blobs. For me this is not OK, to return null as blobMetadata. I agree that we need to find the root cause but all my questions have been left unanswered in this issue and the github PR was closed without my questions been answered there, as well. Also, could you explain to me what is that "eventual consistency"? If this is really the way I agree to put null filtering for the blobs. For now, I do not understand your statement "the blobMetadata could be null -> this could rise NullPointerException but it is okay". Thanks, Encho > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906566#comment-16906566 ] Andrew Gaul commented on JCLOUDS-1504: -- It is possible that {{blobMetadata}} will return {{null}} for for objects recently created or deleted during a {{S3BlobStore.list}} call due to eventual consistency. One possible fix would be to filter out null elements in {{S3BlobStore.list}}. However, it is not obvious to me that this is the behavior you have observed. It might help to examine the logs or share more about your environment. > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887977#comment-16887977 ] Енчо Белезирев commented on JCLOUDS-1504: - Hello [~nacx], I have added a PR for the fix: [https://github.com/jclouds/jclouds/pull/1276] Could you please review it and comment it if you see some problems? > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887117#comment-16887117 ] Енчо Белезирев commented on JCLOUDS-1504: - Hello [~nacx], Is there any update on the issue? There is no progress here and we really need some statement? > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am an SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880991#comment-16880991 ] Енчо Белезирев commented on JCLOUDS-1504: - Hello, As we really need some insight, could someone update the ticket? Also, an explanation will be great. Thanks, Encho > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am an SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874666#comment-16874666 ] Енчо Белезирев commented on JCLOUDS-1504: - Hello, Is there any update here? In my understanding the problem is that when some file is being deleted, when the call to the blobstore is made, the metadata for this deleted file is null. Is there any chance, in such cases the collector of the metadata to not contain the null values? Thanks, Encho > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > Attachments: jclouds-wire.log, jclouds.log > > > Hello, > I am an SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872085#comment-16872085 ] Енчо Белезирев commented on JCLOUDS-1504: - Hi, I am attaching the wire logs. There is something interesting in it. For instance, there are more than one query for certain blob: {code:java} Request: Sending request 782100679: HEAD https://hcp-a0b4ab71-c807-48c5-85b7-797d57d9613e.s3-eu-central-1.amazonaws.com/12bbc932-79c5-4bed-bc9e-a82c6bc4adce HTTP/1.1 Response: Receiving response 782100679: HTTP/1.1 404 Not Found - Request: Sending request 1063600376: HEAD https://hcp-a0b4ab71-c807-48c5-85b7-797d57d9613e.s3-eu-central-1.amazonaws.com/12bbc932-79c5-4bed-bc9e-a82c6bc4adce Response: Receiving response 1063600376: HTTP/1.1 404 Not Found{code} I have used the following code in order to obtain the blob's metadata: {code:java} Iterable modules = ImmutableSet. of(new SLF4JLoggingModule()); // Init BlobStoreContext context = ContextBuilder.newBuilder("aws-s3") .credentials("xxx", "") .modules(modules) .buildView(BlobStoreContext.class); BlobStore blobStore = context.getBlobStore(); final String containerName = ""; PageSet list = blobStore.list(containerName, ListContainerOptions.Builder.withDetails()); for (StorageMetadata resourceMd : list) { System.out.println(resourceMd.getName()); System.out.println(resourceMd.getUserMetadata()); } context.close(); {code} I need the {code:java} ListContainerOptions.Builder.withDetails(){code} because I need the userMetadata which contains some custom info about my blobs. The NullPointerException is happening on this line: {code:java} System.out.println(resourceMd.getName());{code} A little bit of explanation how the blobstore is being read and written: # we are uploading files in it # each file is being deleted after it is processed > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > > Hello, > I am an SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (JCLOUDS-1504) BlobStore.list(container, ListContainerOptions) returns collection containing null elements
[ https://issues.apache.org/jira/browse/JCLOUDS-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871511#comment-16871511 ] Ignasi Barrera commented on JCLOUDS-1504: - We should then try to understand that {{sometimes}} and see what causes that behavior. Can you enable the wire logs and check what does AWS return when you see the null objects? > BlobStore.list(container, ListContainerOptions) returns collection containing > null elements > > > Key: JCLOUDS-1504 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1504 > Project: jclouds > Issue Type: Bug >Reporter: Енчо Белезирев >Priority: Blocker > > Hello, > I am an SAP developer and we are using jclouds for the communication with our > blob stores. > We have issue with the querying of the StorageMetadata for some blobs. We are > using the method from the BlobStore - > {code:java} > PageSet list(String container, > ListContainerOptions options);{code} > However, when we are using the method, we are providing > ListContainerOptions.Builder.withDetails() because we want to take directly > the userMetadata field from the StorageMetadata object and to use it later > on. Here comes the problem, sometimes, when the method is being executed, the > list that is being returned contains null objects. This is a problem because > we want each StorageMetadata to be non-null. > I have validated our entries in the container and it seemed that there are no > suspicious entries in it(which might cause the issue). > Could you share some information what could have caused the issue and > eventually, is there a way to workaround this? > > We are using aws-s3 implementation of the blob store. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)