Hello, Thanks, Ok, I now understand the code better. InodeCache holds 100 Inodes as cache in a ring. When a new inode not in cache is requested, an inode in the cache is flushed, and the new inode is stored in the cache instead. The bug : Race condition : 2 inodes are requested from InodeCache at the same time. Thread 1 requesting inode 1 flush and remove inode 2 from cache. Thread 2 got inode 2 before it was removed from cache by Thread 1, but makes changes after it was removed and flushed by Thread 1. Thead 2 ends, there are no longer references to inode 2, python garbage collect it, and this trigger the bug.
I see 2 possibles solutions : 1/ https://github.com/s3ql/s3ql/pull/196 : The _Inode class keeps a reference to the InodeCache. On __del__, if we encouter the above bug, we flush ourselves. The problem (and I'm not familiar enough with python) : I guess garbage collection could happen at shutdown, when the InodeCache SQL connection is no longer valid. Do you see a way to make this approach work? 2/ (Not coded yet) : The InodeCache is a ring, first in first out. What if we store access time on InodeCache.__getitem__, and the inode to be removed is the most old accessed one? This solution should reduce a lot (but not eliminate) the race condition. What do you think? Finally : I tried to reproduce locally, with a unencrypted, uncompressed local backend, with mass parralel attribute changes, but I was not able to reproduce the bug. Thanks, Nicolas Deschildre On Wednesday, September 9, 2020 at 5:50:19 PM UTC+2 [email protected] wrote: > Hello Nicolas, > > > S3QL somehow manages to delete/garbage collects an _Inode object that is >> dirty (i.e. has an attribute modification that is not persisted in the >> S3QL database) >> > > So, if I understand correctly, since it is a pending modification on a now > deleted inode, this is not really a problem, right? Said otherwise, the > filesystem is not corrupted? [...] > > > No, the inode/file does not have to be deleted. There is a pending > metadata modification (access time, modification time, file size, file > mode) that should have been persisted (written in the Sqlite-Database) but > it did not made it in the database. > File data is not corrupted, but some metadata of your files might not be > in the correct state. > > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/f7d35a3a-14f8-4713-818f-e3c0992a5735n%40googlegroups.com.
