Hi John, the code in LocalBlobStore.putBlob looks like this:
String eTag = storageStrategy.putBlob(containerName, blob);
setBlobAccess(containerName, blobKey, options.getBlobAccess());
I suspect that the object is being removed by another client in between
these calls. Instead FilesystemStorageStrategyImp.putBlob should allow
an options parameter to set the blobAccess on the temporary file which
then gets renamed to the real file name. Would you like to submit a PR
for this?
On Thu, Jan 07, 2021 at 10:09:29PM -0000, John Calcote wrote:
> We use the filesystem (LocalBlobStore) implementation for integration testing
> in our code.
>
> I'm seeing the following behavior in version 2.2.1 which I recently upgraded
> to:
>
> ```
> 2021-01-07 20:03:43.503,7964896021968234 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [0-1048575] - uploading
> 2021-01-07 20:03:43.575,7964896093429743 {} DEBUG
> o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating container
> test-container
> 2021-01-07 20:03:43.575,7964896093478721 {} DEBUG
> o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating directory
> //tmp/16100498223118761388361431415820/test-container
> 2021-01-07 20:03:43.575,7964896093881924 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [1048576-2097151] - checksummed
> checksum=D8ACDD71E006E090F4FBF32CAF0627A3
> 2021-01-07 20:03:43.575,7964896093909591 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [1048576-2097151] - uploading
> ..
> 2021-01-07 20:03:43.576,7964896094167857 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-0] Put blob with key [byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.577,7964896095288960 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-0] Put blob with key [byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.584,7964896102791478 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-1] Put blob with key [byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.597,7964896115564074 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-1] Put blob with key [byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.605,7964896123355487 {} ERROR
> c.p.d.m.j.AbstractUploadCallable [cloud-normalpri-0] [task 1] Upload failed:
> chunk [1048576-2097151] - uploading
> java.lang.RuntimeException: java.nio.file.NoSuchFileException:
> /tmp/16100498223118761388361431415820/test-container/byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3
> at com.google.common.base.Throwables.propagate(Throwables.java:240)
> ~[guava-21.0.jar:?]
> at
> org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.setBlobAccess(FilesystemStorageStrategyImpl.java:694)
> ~[filesystem-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.setBlobAccess(LocalBlobStore.java:409)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:794)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:533)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> ~[guice-3.0.jar:?]
> at com.sun.proxy.$Proxy66.putBlob(Unknown Source) ~[?:?]
> at
> com.hammerspace.csp.rest.client.jclouds.JCloudsCspRestClient$1UploadRunnable.run(JCloudsCspRestClient.java:454)
> [csp-rest-client-4.5.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> Caused by: java.nio.file.NoSuchFileException:
> /tmp/16100498223118761388361431415820/test-container/byHashV1/0001000000000010000020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:218)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:131)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> ~[?:1.8.0_181]
> at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> ~[?:1.8.0_181]
> at java.nio.file.Files.readAttributes(Files.java:1737) ~[?:1.8.0_181]
> at java.nio.file.Files.getPosixFilePermissions(Files.java:2004)
> ~[?:1.8.0_181]
> at
> org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.setBlobAccess(FilesystemStorageStrategyImpl.java:686)
> ~[filesystem-2.2.1.jar:2.2.1]
> ... 14 more
> ```
>
> JClouds seems to be having trouble reading the existing POSIX attributes on
> the file it just wrote (so it can modify them).
>
> Has anyone else seen this, or have there been changes between 2.1 and 2.2.0
> to the filesystem implementation?
--
Andrew Gaul
http://gaul.org/
