Bug#820888: [pkg-cryptsetup-devel] Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Fabian Grünbichler]

On October 10, 2016 9:09:12 PM GMT+02:00, Guilhem Moulin  
wrote:
>Hi Fabian,
>
>On Mon, 10 Oct 2016 at 20:42:00 +0200, Fabian Grünbichler wrote:
>> + [ -r /conf/conf.d/cryptroot ]
>> + exit 0
>
>Looks like /conf/conf.d/cryptroot wasn't generated by the hook script.
>Try adding the option ‘initramfs’ on the 4th column of the crypttab(5)
>entry for each underlying device.  This should force unlocking of these
>devices at initramfs stage.
>
>Hope that helps,
>cheers,

Works like a charm! That was unexpectedly easy, thanks!

Bug#820888: [pkg-cryptsetup-devel] Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Guilhem Moulin]

Hi Fabian,

On Mon, 10 Oct 2016 at 20:42:00 +0200, Fabian Grünbichler wrote:
> + [ -r /conf/conf.d/cryptroot ]
> + exit 0

Looks like /conf/conf.d/cryptroot wasn't generated by the hook script.
Try adding the option ‘initramfs’ on the 4th column of the crypttab(5)
entry for each underlying device.  This should force unlocking of these
devices at initramfs stage.

Hope that helps,
cheers,
-- 
Guilhem.


signature.asc
Description: PGP signature

Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Fabian Grünbichler]

On Mon, 19 Sep 2016 00:43:28 -0500 Richard Laager 
wrote:
> For whatever my opinion is worth... :)
> 
> I see you are using ZFS on top of LVM on top of LUKS. (Or ZFS inside LVM
> inside LUKS, if you prefer that way of looking at it.) While that's
> theoretically something that "should" work, that's not a very practical
> configuration. Specifically, using two volume managers is ill-advised in
> my opinion. I recommend putting swap on ZFS instead of that setup.
> 
> If you really want to get your scenario working, I concur with the
> request for debugging output.
> 
> -- 
> Richard
> 
> 

I am playing around with a slightly different scenario, which is also
not working at all OOTB. Any pointers on where to fix stuff welcome -
I'd be happy to test and figure out patches myself.

first, my basic setup:

- Grub2 with LUKS and ZFS support, taking care of the initial decryption
and loading the kernel and initrd
- complete / on ZFS directly on top of one LUKS-encrypted disk - no
separate /boot , as that is not needed here
- rpool called "rpool" with one member (the mapped decrypted disk), set
up with "-d /dev/disk/by-id"
- / dataset is called "rpool/ROOT/debian", and has the mountpoint
property set to "/"
- crypttab contains a line for the encrypted disk, the crypto device is
named "cryptroot" there (not sure whether this helps at all?)
- initrd generated with set $CRYPTROOT , because it can't detect / from
/etc/fstab (which is empty, all mounts are established by zfs)

$ cat /proc/cmdline
BOOT_IMAGE=(crypto0)/ROOT/debian/@/vmlinuz root=ZFS=rpool/ROOT/debian ro
boot=zfs break=premount cryptops=source=/dev/sda,target=cryptroot

(crypto0) is the device grub "sees" after unlocking the LUKS device with
grub's cryptomount

following the "Debug cryptroot initramfs script", I added the '-x' to
the shebang of the cryptroot script. Unfortunately, running it only
collected the following lines in the log file (note: I was not asked for
a passphrase at this point):

+ PREREQ=cryptroot-prepare
+ . /scripts/functions
+ cmdline_cryptops=
+ unset cmdline_root
+ cat /proc/cmdline
+ opt=ZFS=rpool/ROOT/debian
+ [ -n ]
+ [ -r /conf/conf.d/cryptroot ]
+ exit 0

the /dev nodes are set up correctly, as verified with lsblk (including
the symlinks in by-*)

manually calling "/sbin/cryptsetup luksOpen /dev/disk/by-id/XYZ
cryptroot" followed by "zpool import -d /dev/disk/by-id -R /root rpool"
and pressing ^D a couple of times allows me to boot.

I hacked together a quick script that simply does the above cryptsetup
and zpool dance, which I put into
/etc/initramfs-tools/scripts/local-top/ , this works for now (if I
include a modprobe inbetween, I can boot without any kernel boot
parameters, although the passphrase prompt is not very visible in the
middle of the initrd output). I still have to set $CRYPTSETUP before
generating the initrd with update-initramfs , so I assume automatically
regenerating it on updates would break the setup.

all of this is on sid, with
initramfs-tools 0.125
zfs-initramfs 0.6.5.8-1
cryptsetup 2:1.7.0-2

after an update to the current cryptsetup (2:1.7.2-3) and setting
cryptsetup=y in /etc/crytsetup-initramfs/conf-hook I get the same debug
output and behaviour. uncommenting the /etc/crypttab line and
regenerating the initrd does not change anything.

any pointers appreciated :)

Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Richard Laager]

For whatever my opinion is worth... :)

I see you are using ZFS on top of LVM on top of LUKS. (Or ZFS inside LVM
inside LUKS, if you prefer that way of looking at it.) While that's
theoretically something that "should" work, that's not a very practical
configuration. Specifically, using two volume managers is ill-advised in
my opinion. I recommend putting swap on ZFS instead of that setup.

If you really want to get your scenario working, I concur with the
request for debugging output.

-- 
Richard

Bug#820888: [pkg-cryptsetup-devel] Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Guilhem Moulin]

(Adding the BTS to CC, hopefully you didn't mean to reply to me
privately.)

On Tue, 13 Sep 2016 at 22:47:12 +0200, François Scala wrote:
> I don't understand why do you need an Debian installer package for ZFS.

We test for regressions by making an automated (preseeded), throw-away,
minimal installation for each particular configuration (including disk
partitioning).

Anyway, AFAICT the problem is orthogonal to ZFS per se: it's related to
the handling of the root= kernel parameter in the cryptroot initramfs
boot script.

> This is how I've was able to make this quick fix :
> https://github.com/arcenik/debian-zfs-root/blob/master/d8zlroot.sh (line
> 278)

Still, I don't understand how that can solve your problem.  Granted this
wasn't with a ZFS mount, but passing “root=ZFS=rpool/ROOT/debian-1” and
“cryptops=source=/dev/sda2,target=cryptroot,lvm=vg_sys-lv_zroot” to the
initrd, the NEWROOT variable is properly set to “/dev/mapper/$cryptlvm”
as cmdline_root is left empty.

Could you please enable the debugging in the cryptroot initramfs script?
I know nothing about ZFS, and it'll be faster than me trying to learn it
:-P

-- 
Guilhem.


signature.asc
Description: PGP signature

Bug#820888: [pkg-cryptsetup-devel] Bug#820888: cryptsetup: initramfs cryptroot zfs support

[Guilhem Moulin]

Control: tag -1 moreinfo

Hi François,

On Wed, 13 Apr 2016 at 13:29:41 +0200, François Scala wrote:
> While working on a Debian installation based on ZFS root with LUKS
> encryption (see https://github.com/arcenik/debian-zfs-root) I've encountered
> some problem with the initramfs cryptroot script.

Unfortunately I'm not able to debug this easily because there is
currently no d-i package (udeb) for ZFS :-/

> I think that ${cmdline_root} should be used for NEWROOT only if it begin
> with '/dev' or at least '/' and use '/dev/mapper/${cryptlvm}' if not.

I'm confused: the ‘cmdline_root’ variable is left empty when the value
of the ‘root=’ kernel parameter doesn't start with a slash ‘/’.

root=*)
opt="${opt#root=}"
case $opt in
/*) # Absolute path given. Not lilo major/minor number.
cmdline_root=$opt
;;
*) # lilo major/minor number (See #398957). Ignore
esac

and the ‘NEWROOT’ variable is set to "/dev/mapper/$cryptlvm" when
‘cmdline_root’ is unset or set to the empty string.

Could you try to to enable debugging for the cryptroot initramfs script,
as documented in

  https://wiki.debian.org/CryptsetupDebug#Debug_cryptroot_initramfs_script

and forward us the log file?  The problem might lie somewhere else.

-- 
Guilhem.


signature.asc
Description: PGP signature

Bug#820888: cryptsetup: initramfs cryptroot zfs support

[François Scala]

Package: cryptsetup
Version: 2:1.6.6-5
Severity: normal

Dear Maintainer,

While working on a Debian installation based on ZFS root with LUKS 
encryption (see https://github.com/arcenik/debian-zfs-root) I've 
encountered some problem with the initramfs cryptroot script.


The LUKS is properly opened and the contained LVM volume properly detected.

It then try to detect the filesystem type from the root kernel parameter 
using blkid. But this fail because the root parameter value is not 
recognised by blkid


  root=ZFS=rpool/ROOT/debian-1

I think that ${cmdline_root} should be used for NEWROOT only if it begin 
with '/dev' or at least '/' and use '/dev/mapper/${cryptlvm}' if not.


Regards.

François.

-- Package-specific info:
-- /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=ZFS=rpool/ROOT/debian-1 ro 
boot=zfs cryptops=source=/dev/sda2,target=cryptroot,lvm=vg_sys-lv_zroot


-- /etc/crypttab
# no crypttab

-- /etc/fstab
LABEL=BOOT/bootext4noatime01
LABEL=SWAPnoneswapdefaults00

-- lsmod
Module  Size  Used by
fuse   83350  0
btrfs 863629  0
xor21040  1 btrfs
raid6_pq   95238  1 btrfs
ufs73443  0
qnx4   13036  0
hfsplus   101391  0
hfs53845  0
minix  31387  0
ntfs  194605  0
vfat   17135  0
msdos  17046  0
fat61986  2 vfat,msdos
jfs   172859  0
xfs   779930  0
libcrc32c  12426  1 xfs
crc32c_generic 12656  0
ext4  473801  1
crc16  12343  1 ext4
mbcache17171  1 ext4
jbd2   82514  1 ext4
zfs  2614234  7
zunicode  328546  1 zfs
zcommon39960  1 zfs
znvpair58712  2 zfs,zcommon
spl69541  3 zfs,zcommon,znvpair
zavl   13071  1 zfs
dm_crypt   22595  1
dm_mod 89405  8 dm_crypt
xts12679  1
gf128mul   12970  1 xts
algif_skcipher 17349  0
af_alg 13034  1 algif_skcipher
nfsd  263032  2
auth_rpcgss51211  1 nfsd
oid_registry   12419  1 auth_rpcgss
nfs_acl12511  1 nfsd
nfs   188136  0
lockd  83389  2 nfs,nfsd
fscache45542  1 nfs
sunrpc237402  6 nfs,nfsd,auth_rpcgss,lockd,nfs_acl
ppdev  16782  0
snd_intel8x0   34948  0
joydev 17063  0
snd_ac97_codec118711  1 snd_intel8x0
pcspkr 12595  0
evdev  17445  3
snd_pcm88662  2 snd_ac97_codec,snd_intel8x0
serio_raw  12849  0
snd_timer  26614  1 snd_pcm
snd65244  4 
snd_ac97_codec,snd_intel8x0,snd_timer,snd_pcm

battery13356  0
soundcore  13026  1 snd
parport_pc 26300  0
parport35749  2 ppdev,parport_pc
processor  28221  0
video  18096  0
button 12944  0
ac 12715  0
ac97_bus   12510  1 snd_ac97_codec
thermal_sys27642  2 video,processor
i2c_piix4  20864  0
i2c_core   46012  1 i2c_piix4
autofs435529  2
squashfs   43856  1
loop   26605  3
aufs  199535  390
nls_utf8   12456  4
isofs  38965  1
hid_generic12393  0
usbhid 44460  0
hid   102264  2 hid_generic,usbhid
sd_mod 44356  3
crc_t10dif 12431  1 sd_mod
sg 29973  0
sr_mod 21903  1
cdrom  47424  1 sr_mod
crct10dif_generic  12581  1
crct10dif_common   12356  2 crct10dif_generic,crc_t10dif
ata_generic12490  0
ohci_pci   12808  0
crc32c_intel   21809  2
ata_piix   33592  1
ahci   4  2
libahci27158  1 ahci
psmouse99249  0
ohci_hcd   42982  1 ohci_pci
ehci_pci   12512  0
ehci_hcd   69837  1 ehci_pci
libata177508  4 ahci,libahci,ata_generic,ata_piix
scsi_mod  191405  4 sg,libata,sd_mod,sr_mod
usbcore   195468  5 ohci_hcd,ohci_pci,ehci_hcd,ehci_pci,usbhid
usb_common 12440  1 usbcore
e1000 122545  0


-- System Information:
Debian Release: 8.4
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd

Versions of packages cryptsetup depends on:
ii  cryptsetup-bin