Hi Andrew,
Your theory about another client deleting the file cannot be correct. This is
an integration test where a single test is running - the container is created
in /tmp/<some-random-directory>/test-container. The test creates the test
container in the setup method, uploads several files, manipulates some data,
and then deletes the entire container in the teardown method.
This is the very first file added, and the test-container itself was created
milliseconds earlier. There are no other clients running at this time. If you
look back at the log snippet I pasted, you'll see that the log line
2021-01-07 20:03:43.576,7964896094167857 {} DEBUG o.j.b.c.LocalBlobStore
[cloud-normalpri-0] Put blob with key [byHashV1/0001000000000010000020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to container
[test-container]
which is logged by jclouds filesystem impl is logged four times in a row
(because jclouds is retrying under the covers for us, as we use Payload style
payloads, which are retriable). Only the last error is logged ultimately, but
it's clear this error is happening at least four times in quick succession
before jclouds gives up and throws the exception (which we log).
Just prior to this log line, jclouds logs the creation of the container and the
test-container directory.
We upgraded from 2.2.0 to 2.2.1 recently and this started happening. The test
has been in place for years, without seeing this issue. Suddenly, after
upgrading, it starts happening.
Thanks,
John
