Hi
On Tue, 11 Feb 2025, Marco Felsch wrote: > Hi all, > > as written in the subject we do see an odd dm-integrity behaviour during > the journal replay step. > > First things first, a short introduction to our setup: > - Linux v6.8 > - We do have a dm-integrity+dm-crypt setup to provide an authenticated > encrypted ext4-based rw data partition. > - The setup is done with a custom script [1] since we are making use of > the kernel trusted-keys infrastructure which isn't supported by LUKS2 > at the moment. > - The device has no power failsafe e.g. hard power-cuts can appear. > Therefore we use the dm-integrity J(ournal) mode. > - The storage backend is an eMMC with 512Byte block size. Could you retest it with an eMMC or SDCARD from a different vendor, just to test if it is hardware issue? > - We use the dm-integrity 4K block size option to reduce the > tag/metadata overhead. > > From time to time we do see "AEAD ERROR"s [2] while fsck tries to repair > the filesystem which of course abort the fsck run. > After a while within the rescue shell So, when you run fsck again from the rescue shell (without deactivating and activating the device), the bug goes away? What is the time interval after which the bug goes away? Could you upload somewhere the image of the eMMC storage when the bug happens and send me a link to it, so that I can look what kind of corruption is there? > and a following reboot the fsck > run on the same file system doesn't trigger any "AEAD ERROR". > > The dm-integrity table is added twice [1] since we gather the > provided_data_sectors information from the first call. I know that this > isn't optimal. The provided_data_sectors should be stored and not > gathered again but this shouldn't bother a system with a valid > dm-integrity superblock already written. That should work fine - there is no problem with activating the device twice. > To debug the issue we uncommented the "#define DEBUG_PRINT" and noticed > that the replay is happening twice [2] albeit this should be a > synchronous operation and once the dm resume returned successfully the > replay should be done. The replay happens twice because you activate it twice - this should be harmless because each replay replays the same data - there is "replaying 364 sections, starting at 143, commit seq 2" twice in the log. > We also noticed that once we uncommented "#define DEBUG_PRINT" it was > harder to trigger the issue. > > We also checked the eMMC bus to see if the CMD23 is sent correctly (with > the reliable write bit set) in case of a FUA request which is the case. > So now with the above knowledge we suspect the replay path to be not > synchronous, the dm resume is returning too early, while not all of the > writes kicked off by copy_from_journal having reached the storage. There is "struct journal_completion comp" in do_journal_write and "wait_for_completion_io(&comp.comp);" at the end of do_journal_write - that should wait until all I/O submitted by copy_from_journal finishes. > Maybe you guys have some pointers we could follow or have an idea of > what may go wrong here. > > Regards, > Marco I read through the dm-integrity source code and I didn't find anything. There are two dm-crypt bugs that could cause your problems - see the upstream commits 8b8f8037765757861f899ed3a2bfb34525b5c065 and 9fdbbdbbc92b1474a87b89f8b964892a63734492. Please, backport these commits to your kernel and re-run the tests. Mikulas
