Re: cannot access pass device from within jail
> On Dec 17, 2017, at 4:48 PM, Warner Losh wrote: > > Sorry to top post. The problem did turn out to be securelevel. We found > this by doing > > dtrace -n 'fbt::securelevel_gt:return {print(args[1];)}' > > Though you could also replace securelevel_gt with passopen to see it was in > passopen too... On the host we ran: [root@r710-01:~] # sudo dtrace -n 'fbt::securelevel_gt:return {print(args[1]);}' dtrace: description 'fbt::securelevel_gt:return ' matched 1 probe CPU IDFUNCTION:NAME 21 50269securelevel_gt:return int 0x1 In the jail: mtx -f /dev/pass7 status Based on the dtrace output, I again checked securelevel in the jail: [dan@bacula-sd-02] $ sysctl kern.securelevel kern.securelevel: 2 WTF? I'd already checked that as seen at https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007614.html Oh wait. The above URL refers to bacula-sd-01, not bacula-sd-02. Ouch. User error... me. After going back into the host, I set: $ sudo iocage set securelevel=1 bacula-sd-02 Property: securelevel has been updated to 1 After restarting the jail and getting back into it: [root@bacula-sd-02 ~]# sysctl kern.securelevel kern.securelevel: 1 [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status Storage Changer /dev/pass7:2 Drives, 47 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = 01L4 Success. Sorry for the noise which would have been reduced if I'd gotten my sysctl on the right host. Thank you all. > > Warner > > On Sun, Dec 17, 2017 at 2:08 PM, Dan Langille wrote: > >>> On Dec 17, 2017, at 4:04 PM, Warner Losh wrote: >>> >>> What's the permissions of /dev/xpt0 in the jail? If it's not there I know >>> at least camcontrol won't work. I've not used mtx, so I can't say if it's >>> affected too or not. >> >> I have tried both with and without xpt0. When I tried, it was: >> >> # ls -l /dev/xpt0 >> crw--- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0 >> >>> >>> However, looking at the truss output: >>> >>> openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not >>> permitted' >>> suggests something other than the canonical xpt0 issue else is going on. >> If >>> we look at passopen in cam, I can see two exit paths: >>> >>> error = securelevel_gt(td->td_ucred, 1); if (error != 0) {... >>> return error; } >>> securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ? >>> EPERM : 0);" which might be possible. What's the securelevel of the jail? >>> Maybe this is going on somehow? >> >> >> On the host: >> >> $ sysctl kern.securelevel >> kern.securelevel: -1 >> >> >> On the jail: >> >> $ sysctl kern.securelevel >> kern.securelevel: -1 >> >>> >>> The second is basically >>> if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return >>> EPERM; } >>> which isn't happening because of the O_RDWR in the truss output. >>> >>> The other possibility is that something above the pass driver is doing >> the >>> check. I've not looked at that code path yet, buy you can see if it's >>> making it to passopen() with dtrace and checking its return value. I >> don't >>> see anything in how we register the device, though, that would suggest >>> filtering it in jails. >>> >>> Warner >>> >>> On Sun, Dec 17, 2017 at 12:52 PM, Dan Langille wrote: >>> Hello, What suggestions do you have for where I should look next? I'm happy to start installing various builds of FreeBSD in order to track down which commit caused this. I'm trying to access a tape library from within a jail running on a FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). pass(4) devices (i.e. the tape changer ch0) are not working. This >> morning I posted to -scsi@: https://lists.freebsd.org/ >> pipermail/freebsd-scsi/2017- December/007608.html The device appears in the jail and has appropriate permissions. This access was granted via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 The permissions in the jail: [root@bacula-sd-02 ~]# ls -l /dev/pass7 crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 The command in the jail: [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status cannot open SCSI device '/dev/pass7' - Operation not permitted Here is the truss output of the command in question: https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe Thank you. -- Dan Langille - BSDCan / PGCon d...@langille.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@ >> freebsd.org" >>> ___ >>> freebsd-curren
Re: cannot access pass device from within jail
Sorry to top post. The problem did turn out to be securelevel. We found this by doing dtrace -n 'fbt::securelevel_gt:return {print(args[1];)}' Though you could also replace securelevel_gt with passopen to see it was in passopen too... Warner On Sun, Dec 17, 2017 at 2:08 PM, Dan Langille wrote: > > On Dec 17, 2017, at 4:04 PM, Warner Losh wrote: > > > > What's the permissions of /dev/xpt0 in the jail? If it's not there I know > > at least camcontrol won't work. I've not used mtx, so I can't say if it's > > affected too or not. > > I have tried both with and without xpt0. When I tried, it was: > > # ls -l /dev/xpt0 > crw--- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0 > > > > > However, looking at the truss output: > > > > openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not > > permitted' > > suggests something other than the canonical xpt0 issue else is going on. > If > > we look at passopen in cam, I can see two exit paths: > > > >error = securelevel_gt(td->td_ucred, 1); if (error != 0) {... > > return error; } > > securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ? > > EPERM : 0);" which might be possible. What's the securelevel of the jail? > > Maybe this is going on somehow? > > > On the host: > > $ sysctl kern.securelevel > kern.securelevel: -1 > > > On the jail: > > $ sysctl kern.securelevel > kern.securelevel: -1 > > > > > The second is basically > >if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return > > EPERM; } > > which isn't happening because of the O_RDWR in the truss output. > > > > The other possibility is that something above the pass driver is doing > the > > check. I've not looked at that code path yet, buy you can see if it's > > making it to passopen() with dtrace and checking its return value. I > don't > > see anything in how we register the device, though, that would suggest > > filtering it in jails. > > > > Warner > > > > On Sun, Dec 17, 2017 at 12:52 PM, Dan Langille wrote: > > > >> Hello, > >> > >> What suggestions do you have for where I should look next? I'm happy to > >> start installing various builds of FreeBSD in order to track down which > >> commit caused this. > >> > >> I'm trying to access a tape library from within a jail running on a > >> FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). > >> > >> pass(4) devices (i.e. the tape changer ch0) are not working. This > morning > >> I posted to -scsi@: https://lists.freebsd.org/ > pipermail/freebsd-scsi/2017- > >> December/007608.html > >> > >> The device appears in the jail and has appropriate permissions. This > >> access was granted > >> via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 > >> > >> The permissions in the jail: > >> > >> [root@bacula-sd-02 ~]# ls -l /dev/pass7 > >> crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 > >> > >> The command in the jail: > >> > >> [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status > >> cannot open SCSI device '/dev/pass7' - Operation not permitted > >> > >> Here is the truss output of the command in question: > >> https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe > >> > >> Thank you. > >> > >> -- > >> Dan Langille - BSDCan / PGCon > >> d...@langille.org > >> > >> > >> ___ > >> freebsd-current@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-current > >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@ > freebsd.org" > >> > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@ > freebsd.org" > > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
> On Dec 17, 2017, at 4:04 PM, Warner Losh wrote: > > What's the permissions of /dev/xpt0 in the jail? If it's not there I know > at least camcontrol won't work. I've not used mtx, so I can't say if it's > affected too or not. I have tried both with and without xpt0. When I tried, it was: # ls -l /dev/xpt0 crw--- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0 > > However, looking at the truss output: > > openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not > permitted' > suggests something other than the canonical xpt0 issue else is going on. If > we look at passopen in cam, I can see two exit paths: > >error = securelevel_gt(td->td_ucred, 1); if (error != 0) {... > return error; } > securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ? > EPERM : 0);" which might be possible. What's the securelevel of the jail? > Maybe this is going on somehow? On the host: $ sysctl kern.securelevel kern.securelevel: -1 On the jail: $ sysctl kern.securelevel kern.securelevel: -1 > > The second is basically >if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return > EPERM; } > which isn't happening because of the O_RDWR in the truss output. > > The other possibility is that something above the pass driver is doing the > check. I've not looked at that code path yet, buy you can see if it's > making it to passopen() with dtrace and checking its return value. I don't > see anything in how we register the device, though, that would suggest > filtering it in jails. > > Warner > > On Sun, Dec 17, 2017 at 12:52 PM, Dan Langille wrote: > >> Hello, >> >> What suggestions do you have for where I should look next? I'm happy to >> start installing various builds of FreeBSD in order to track down which >> commit caused this. >> >> I'm trying to access a tape library from within a jail running on a >> FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). >> >> pass(4) devices (i.e. the tape changer ch0) are not working. This morning >> I posted to -scsi@: https://lists.freebsd.org/pipermail/freebsd-scsi/2017- >> December/007608.html >> >> The device appears in the jail and has appropriate permissions. This >> access was granted >> via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 >> >> The permissions in the jail: >> >> [root@bacula-sd-02 ~]# ls -l /dev/pass7 >> crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 >> >> The command in the jail: >> >> [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status >> cannot open SCSI device '/dev/pass7' - Operation not permitted >> >> Here is the truss output of the command in question: >> https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe >> >> Thank you. >> >> -- >> Dan Langille - BSDCan / PGCon >> d...@langille.org >> >> >> ___ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" >> > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
What's the permissions of /dev/xpt0 in the jail? If it's not there I know at least camcontrol won't work. I've not used mtx, so I can't say if it's affected too or not. However, looking at the truss output: openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not permitted' suggests something other than the canonical xpt0 issue else is going on. If we look at passopen in cam, I can see two exit paths: error = securelevel_gt(td->td_ucred, 1); if (error != 0) {... return error; } securelevel_gt is just "return (cr->cr_prison->pr_securelevel > level ? EPERM : 0);" which might be possible. What's the securelevel of the jail? Maybe this is going on somehow? The second is basically if (((flags & FWRITE) == 0) || ((flags & FREAD) == 0)) {... return EPERM; } which isn't happening because of the O_RDWR in the truss output. The other possibility is that something above the pass driver is doing the check. I've not looked at that code path yet, buy you can see if it's making it to passopen() with dtrace and checking its return value. I don't see anything in how we register the device, though, that would suggest filtering it in jails. Warner On Sun, Dec 17, 2017 at 12:52 PM, Dan Langille wrote: > Hello, > > What suggestions do you have for where I should look next? I'm happy to > start installing various builds of FreeBSD in order to track down which > commit caused this. > > I'm trying to access a tape library from within a jail running on a > FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). > > pass(4) devices (i.e. the tape changer ch0) are not working. This morning > I posted to -scsi@: https://lists.freebsd.org/pipermail/freebsd-scsi/2017- > December/007608.html > > The device appears in the jail and has appropriate permissions. This > access was granted > via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 > > The permissions in the jail: > > [root@bacula-sd-02 ~]# ls -l /dev/pass7 > crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 > > The command in the jail: > > [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status > cannot open SCSI device '/dev/pass7' - Operation not permitted > > Here is the truss output of the command in question: > https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe > > Thank you. > > -- > Dan Langille - BSDCan / PGCon > d...@langille.org > > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
On Sun, Dec 17, 2017 at 03:49:15PM -0500, Dan Langille wrote: > > On Dec 17, 2017, at 3:37 PM, Konstantin Belousov > > wrote: > > Does it work to access the pass device from host using host' /dev ? > > Yes, it does. see "This command on the host" at > https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007610.html Ok. > > > Same question for the host access using the nodes of the jailed devfs mount. > > I didn't try that, but I will soon. To be clear, does this command on the > host look like what you have in mind? > > mtx -f /usr/jails/bacula-sd-02/dev/pass7 status I do not know. Check with truss which node gets accessed. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
> On Dec 17, 2017, at 3:37 PM, Konstantin Belousov wrote: > > On Sun, Dec 17, 2017 at 02:52:12PM -0500, Dan Langille wrote: >> Hello, >> >> What suggestions do you have for where I should look next? I'm happy to >> start installing various builds of FreeBSD in order to track down which >> commit caused this. >> >> I'm trying to access a tape library from within a jail running on a FreeBSD >> 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). >> >> pass(4) devices (i.e. the tape changer ch0) are not working. This morning I >> posted to -scsi@: >> https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007608.html >> >> The device appears in the jail and has appropriate permissions. This access >> was granted >> via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 >> >> The permissions in the jail: >> >> [root@bacula-sd-02 ~]# ls -l /dev/pass7 >> crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 >> >> The command in the jail: >> >> [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status >> cannot open SCSI device '/dev/pass7' - Operation not permitted >> >> Here is the truss output of the command in question: >> https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe > > Does it work to access the pass device from host using host' /dev ? Yes, it does. see "This command on the host" at https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007610.html > Same question for the host access using the nodes of the jailed devfs mount. I didn't try that, but I will soon. To be clear, does this command on the host look like what you have in mind? mtx -f /usr/jails/bacula-sd-02/dev/pass7 status -- Dan Langille - BSDCan / PGCon d...@langille.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
> On Dec 17, 2017, at 3:37 PM, Bjoern A. Zeeb > wrote: > > On 17 Dec 2017, at 19:52, Dan Langille wrote: > >> Hello, >> >> What suggestions do you have for where I should look next? I'm happy to >> start installing various builds of FreeBSD in order to track down which >> commit caused this. >> >> I'm trying to access a tape library from within a jail running on a FreeBSD >> 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). >> >> pass(4) devices (i.e. the tape changer ch0) are not working. This morning I >> posted to -scsi@: >> https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007608.html >> >> The device appears in the jail and has appropriate permissions. This access >> was granted >> via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 >> >> The permissions in the jail: >> >> [root@bacula-sd-02 ~]# ls -l /dev/pass7 >> crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 >> >> The command in the jail: >> >> [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status >> cannot open SCSI device '/dev/pass7' - Operation not permitted >> >> Here is the truss output of the command in question: >> https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe > > > You don’t by any chance have a securelevel > 1 set for that jail? On the host: $ sysctl kern.securelevel kern.securelevel: -1 On the jail: $ sysctl kern.securelevel kern.securelevel: -1 Thank you -- Dan Langille - BSDCan / PGCon d...@langille.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
On 17 Dec 2017, at 19:52, Dan Langille wrote: Hello, What suggestions do you have for where I should look next? I'm happy to start installing various builds of FreeBSD in order to track down which commit caused this. I'm trying to access a tape library from within a jail running on a FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). pass(4) devices (i.e. the tape changer ch0) are not working. This morning I posted to -scsi@: https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007608.html The device appears in the jail and has appropriate permissions. This access was granted via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 The permissions in the jail: [root@bacula-sd-02 ~]# ls -l /dev/pass7 crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 The command in the jail: [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status cannot open SCSI device '/dev/pass7' - Operation not permitted Here is the truss output of the command in question: https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe You don’t by any chance have a securelevel > 1 set for that jail? /bz ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cannot access pass device from within jail
On Sun, Dec 17, 2017 at 02:52:12PM -0500, Dan Langille wrote: > Hello, > > What suggestions do you have for where I should look next? I'm happy to start > installing various builds of FreeBSD in order to track down which commit > caused this. > > I'm trying to access a tape library from within a jail running on a FreeBSD > 11.1 host. sa(4) devices are working (e.g. I can rewind nsa0). > > pass(4) devices (i.e. the tape changer ch0) are not working. This morning I > posted to -scsi@: > https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007608.html > > The device appears in the jail and has appropriate permissions. This access > was granted > via /etc/devfs.rules using the same approach I used for FreeBSD 10.3 > > The permissions in the jail: > > [root@bacula-sd-02 ~]# ls -l /dev/pass7 > crw--- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 > > The command in the jail: > > [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status > cannot open SCSI device '/dev/pass7' - Operation not permitted > > Here is the truss output of the command in question: > https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe Does it work to access the pass device from host using host' /dev ? Same question for the host access using the nodes of the jailed devfs mount. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"