Re: [dm-devel] can we fix some dm snapshot crashes please? :)
07.05.2021 18:20, Mike Snitzer wrote: On Fri, May 07 2021 at 10:10P -0400, Michael Tokarev wrote: 07.05.2021 15:31, Zdenek Kabelac wrote: Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a): ... sz=$(blockdev --getsize /dev/loop0) dmsetup create base --table "0 $sz snapshot-origin /dev/loop0" # and now the crash mkfs.ext4 /dev/mapper/base Yes reproducible - can you please open BZ report here: https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper Ok, will do, thanks! Thanks. But when you do, please tone down the lamenting of how slow snapshot crashes have been fixed -- or don't. But that line of rhetoric shows you're full of it. That's a joke really - note the smile at the end of my statement. However this is true, - this particular crash has been here for a long time, someone else asked about it earlier too, and I asked about it here 1.5 years ago when trying snapshots the previous time (actually when just trying to _understand_ how to do it properly). Not really, - it is just a joke, and I know full well how it all is done. .. Yes, it is dangerous to stab in the dark like you clearly are doing. I'd LOVE to know how to do it properly. Unfortunately I found no information about it. There's one usage scenario outlined in a few places which talks about the topic, all are doing it similar to how LVM is doing it. But the thing is that there are actually numerous usage scenarios for the feature, it definitely can be used numerous ways, - the few mentions of dm snapshots offers one particular scenario but does not show even the basic rules. There's an email (on this list IIRC) from about a decade ago which defines a similar scenario and which - apparently - is the only actual source of information about the whole thing, and it again does not describe the game _rules_ :) What you see as "I'm full of it" is a laugh actually - there's nether a defined rules (for which many users asked in various places), and there are crashes, and each time one points to a crash someone points to "you're using a wrong rules". Which turns into a dead circle... :) My actual intention is two-fold: - besides trying to make the thing less dangerous, I'd *love* to make some more clear docs, describing not only one particular scenario but the actual rules, so to say. There's a significant gap between developers understanding how this layer and this particular feature of this layer works, and the rest of the world. Somehow this gap seems to be a bit more significant than it is for many other areas (I'm a (positive! :) ) sysadmin and system developer for 20+ years and know how things are working). Just to give an example: I for one was puzzled by dmsetup reload - why it is not actually replacing the table *and* returning success?? I started digging into kernel sources and doing some experiments, and just by a chance discovered that the table *will* be replaced by the next "dmsetup resume", - even for a non-suspended device! This might be an obvious as an air thing for people who know some internals, but it is non-obvious even for some users doing things in this area for years. I asked several friends of mine who worked with dmsetup and definitely knows it - no, they never *tried* to reload a table without explicit suspend-resume, so the operation is always 3-fold instead of single. While the docs mentions it somehow, one should have the question or a guess to notice or to find this in the docs. Also the docs mentions, like, "should be be suspended or else something can go wrong if the device is in active use" - note the "if". So I assumed that nothing can go wrong if the device is not in use. And finally, maybe this is just a somewhat-uneasily designed software when it requires the same 3 operations for a single operation? Back to this snapshot-origin thing, to its *usage*. The usual example given in a few places says: 1. create a linear 1:1 mapping of the base volume 2. mount it 3. create a copy of the mapping from step 1 4. suspend the mapping from step 1 5. create snapshot device of the device from step 3 6. *replace* the original mapping from step 1 with snapshot-origin device based on the mapping from step 3 7. resume the snapshot-origin device 8. the result is that the filesystem mounted in step 2 is now based on the snapshot-origin instead of the linear mapping. (I can have something backwards or slightly wrong but the concept should be the same). This gives tons of questions. The first question is why we need to *replace* the table and why we need to mount it before replacing? Why can't we mount (if it needs to be mounted) snapshot-origin device (if that's the thing which gets mounted in the end) instead of using snapshot-origin directly? ...hmm. After re-reading the docs in the gentoo wiki, I think I see the issue in my (mis)understanding, even if it is not explicitly clear. It *might* be the snapshot-origin is not a separate device which is useful
Re: [dm-devel] can we fix some dm snapshot crashes please? :)
On Fri, May 07 2021 at 10:10P -0400, Michael Tokarev wrote: > 07.05.2021 15:31, Zdenek Kabelac wrote: > > Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a): > ... > > > sz=$(blockdev --getsize /dev/loop0) > > > dmsetup create base --table "0 $sz snapshot-origin /dev/loop0" > > > # and now the crash > > > mkfs.ext4 /dev/mapper/base > > > Yes reproducible - can you please open BZ report here: > > > > https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper > > Ok, will do, thanks! Thanks. But when you do, please tone down the lamenting of how slow snapshot crashes have been fixed -- or don't. But that line of rhetoric shows you're full of it. > > I assume you are aware you are trying to using snapshot target in wrong way, > > but it should not be crashing kernel. > > Actually I'm not aware about the wrong way. The documentation is a bit lacking > in this area. There are a few (maybe 2 or 3) guides describing the steps to > do, > most involves first using an 1:1 linear table, next replacing it on-the-fly > with > snapshot-origin table, but nowhere it is told that I can't create > snapshot-origin > directly, or that I need at least one snapshot device, or that the origin of > the > snapshot-origin table should be a dm device, or many many other variations. > > I just tried to create a snapshot to be able to revert changes if something > goes > wrong, and since I don't use lvm I thought that plain dmsetup should do, but > discovered that it doesn't quite work and it actually is a dangerous thing to > use.. :) Yes, it is dangerous to stab in the dark like you clearly are doing. -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] can we fix some dm snapshot crashes please? :)
07.05.2021 15:31, Zdenek Kabelac wrote: Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a): ... sz=$(blockdev --getsize /dev/loop0) dmsetup create base --table "0 $sz snapshot-origin /dev/loop0" # and now the crash mkfs.ext4 /dev/mapper/base Yes reproducible - can you please open BZ report here: https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper Ok, will do, thanks! I assume you are aware you are trying to using snapshot target in wrong way, but it should not be crashing kernel. Actually I'm not aware about the wrong way. The documentation is a bit lacking in this area. There are a few (maybe 2 or 3) guides describing the steps to do, most involves first using an 1:1 linear table, next replacing it on-the-fly with snapshot-origin table, but nowhere it is told that I can't create snapshot-origin directly, or that I need at least one snapshot device, or that the origin of the snapshot-origin table should be a dm device, or many many other variations. I just tried to create a snapshot to be able to revert changes if something goes wrong, and since I don't use lvm I thought that plain dmsetup should do, but discovered that it doesn't quite work and it actually is a dangerous thing to use.. :) Thanks! /mjt -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel
Re: [dm-devel] can we fix some dm snapshot crashes please? :)
Dne 07. 05. 21 v 12:31 Michael Tokarev napsal(a): Hi! I asked this 1.5 years ago, but go no replies. The thing is that we have a few bugs in kernel in snapshot-related area for years, maybe for decades, and the bugs are still there and it is still trivial to crash the kernel is you're not very-very-very careful. This might be due to user error ofcourse, but this still is definitely NOT a valid reason for the crashes. For example, here it goes for snapshot-origin dm target. Just create a snapshot-origin and do some activity on it, it is a 2-line reproducer: # first create a test device: it can be anything # here we use a 100-MB loop device truncate --size=100M base losetup /dev/loop0 base # now create the snapshot-origin on it sz=$(blockdev --getsize /dev/loop0) dmsetup create base --table "0 $sz snapshot-origin /dev/loop0" # and now the crash mkfs.ext4 /dev/mapper/base This crashes instantly. Different kernels crashed a bit differently, I tried even some 3.x kernels. But the result is the same - crash. Here's an example from 5.10 kernel: Hi Yes reproducible - can you please open BZ report here: https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper I assume you are aware you are trying to using snapshot target in wrong way, but it should not be crashing kernel. Regards Zdeneek <>-- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel
[dm-devel] can we fix some dm snapshot crashes please? :)
Hi!
I asked this 1.5 years ago, but go no replies.
The thing is that we have a few bugs in kernel in snapshot-related area
for years, maybe for decades, and the bugs are still there and it is still
trivial to crash the kernel is you're not very-very-very careful.
This might be due to user error ofcourse, but this still is definitely
NOT a valid reason for the crashes.
For example, here it goes for snapshot-origin dm target. Just create
a snapshot-origin and do some activity on it, it is a 2-line reproducer:
# first create a test device: it can be anything
# here we use a 100-MB loop device
truncate --size=100M base
losetup /dev/loop0 base
# now create the snapshot-origin on it
sz=$(blockdev --getsize /dev/loop0)
dmsetup create base --table "0 $sz snapshot-origin /dev/loop0"
# and now the crash
mkfs.ext4 /dev/mapper/base
This crashes instantly. Different kernels crashed a bit differently,
I tried even some 3.x kernels. But the result is the same - crash.
Here's an example from 5.10 kernel:
[ 89.661594] [ cut here ]
[ 89.663789] kernel BUG at block/bio.c:1473!
[ 89.665624] invalid opcode: [#1] SMP PTI
[ 89.669108] CPU: 0 PID: 264 Comm: mkfs.ext4 Not tainted 5.10.0-6-amd64 #1
Debian 5.10.28-1
[ 89.672551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.14.0-2 04/01/2014
[ 89.675987] RIP: 0010:bio_split+0x74/0x80
[ 89.677744] Code: 89 c7 e8 ff 5e 03 00 41 8b 74 24 28 48 89 ef e8 e2 f5 ff ff f6 45 15 01 74 08 66 41 81 4c 24 14 00 01 4c 89 e0 5b 5d 41 5c c3
<0f> 0b 0f 0b 0f 0b 45 31 e4 eb ed 90 0f 1f 44 00 00 39 77 28 76 05
[ 89.686181] RSP: 0018:b248c026bb30 EFLAGS: 00010246
[ 89.688347] RAX: 0008 RBX: RCX: 8bd5025f7d80
[ 89.691201] RDX: 0c00 RSI: RDI: 8bd502031780
[ 89.694089] RBP: R08: 0019a1b717a8 R09:
[ 89.696975] R10: 8bd5341fc600 R11: 8bd5341fc658 R12: 8bd5024a0558
[ 89.699935] R13: 8bd5024a R14: 8bd502031780 R15: 8bd502383c80
[ 89.704128] FS: 7f0a33236780() GS:8bd53ea0()
knlGS:
[ 89.707641] CS: 0010 DS: ES: CR0: 80050033
[ 89.710162] CR2: 7f0a31db2000 CR3: 02ab4000 CR4: 06f0
[ 89.713071] Call Trace:
[ 89.714247] dm_submit_bio+0x35d/0x440 [dm_mod]
[ 89.716230] submit_bio_noacct+0xf8/0x420
[ 89.719430] ? bio_add_page+0x62/0x90
[ 89.721073] submit_bh_wbc+0x16a/0x190
[ 89.722702] __block_write_full_page+0x1fa/0x460
[ 89.724767] ? bdev_evict_inode+0xc0/0xc0
[ 89.726523] ? block_invalidatepage+0x150/0x150
[ 89.728446] __writepage+0x17/0x60
[ 89.730010] write_cache_pages+0x186/0x3d0
[ 89.731789] ? __wb_calc_thresh+0x120/0x120
[ 89.734842] generic_writepages+0x4c/0x80
[ 89.736830] do_writepages+0x34/0xc0
[ 89.738390] ? __fsnotify_parent+0xe7/0x2d0
[ 89.740196] __filemap_fdatawrite_range+0xc5/0x100
[ 89.742245] file_write_and_wait_range+0x61/0xb0
[ 89.744219] blkdev_fsync+0x17/0x40
[ 89.745777] __x64_sys_fsync+0x34/0x60
[ 89.747446] do_syscall_64+0x33/0x80
[ 89.749341] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 89.753171] RIP: 0033:0x7f0a320b37a0
[ 89.754777] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 83 3d 69 cf 2b 00 00 75 10 b8 4a 00 00 00 0f 05
<48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 4e 3d 01 00 48 89 04 24
[ 89.762134] RSP: 002b:7ffdb7a55158 EFLAGS: 0246 ORIG_RAX:
004a
[ 89.765245] RAX: ffda RBX: RCX: 7f0a320b37a0
[ 89.770073] RDX: 0400 RSI: 55797fd3e620 RDI: 0003
[ 89.773025] RBP: 55797fd3e510 R08: 04800800 R09: 7f0a32e08c40
[ 89.775864] R10: 04800800 R11: 0246 R12: 7ffdb7a551c0
[ 89.778680] R13: 7ffdb7a551c8 R14: 55797fd3e2e0 R15:
[ 89.781984] Modules linked in: dm_snapshot dm_bufio dm_mod loop hid_generic usbhid hid uhci_hcd ehci_hcd virtio_net net_failover ppdev failover
joydev usbcore parport_pc psmouse evdev pcspkr serio_raw parport ata_generic floppy i2c_piix4 virtio_pci sg usb_common ata_piix virtio_ring virtio
button qemu_fw_cfg ip_tables x_tables autofs4 crc32c_generic ext4 crc16 mbcache jbd2 sd_mod t10_pi crc_t10dif crct10dif_common ahci libahci libata
scsi_mod
[ 89.799301] ---[ end trace 1aa4a3cf509dc9b7 ]---
(block/bio.c:1473 is this:
struct bio *bio_split(struct bio *bio, int sectors,
gfp_t gfp, struct bio_set *bs)
{
struct bio *split;
BUG_ON(sectors <= 0); < here
BUG_ON(sectors >= bio_sectors(bio));
)
I don't know what the problem with this is, but the kernel should
definitely not crash this easy, taking down whole system with it...
The same happens when at least one snapshot target is also associated
with this loop device, or when
