Re: RFC: nfsd in a vnet jail
On 2022-12-01 17:32, Rick Macklem wrote: On Thu, Dec 1, 2022 at 8:23 AM Chris wrote: On 2022-11-29 16:21, Rick Macklem wrote: > On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson wrote: > >> Keep the global variables as defaults that apply to all nfsds and allow >> (at least some subset) to be overridden inside the net jails if some things >> need to be changed from the defaults? >> >> This is pretty much a reply to one of the posts selected at random, > but I thought that better than starting a new email thread. > > bz@ and asomers@ have both asked about running mountd within a vnet prison > (one via offlist email and the other on phabricator). > > I think it is worth discussing here... > mountd (rightly or wrongly) does two distinctly different things: > 1 - It pushes the exports into the kernel via nmount() so they > can be hung off of the "struct mount" for a file system's > mount point. > --> This can only work for file system mount points and can > only be done once for any given file system mount point. > > At this time, I have it done once globally outside of the prisons. > The alternative I can see is doing it within each prison, but I > think that would require that each prison have its own file system(s). > (ie. The prison's root would always be a file system mount point.) > > 2 - It handles RPC Mount protocol requests from NFSv3 clients. This one > is NFSv3 specific, which is why I have done this NFSv4 only at > this time. To do this, it must be able to register with rpcbind, > and I have no idea if running rpcbind in a vnet jail is practical. > > Enforcing the use for separate file systems for each jail also makes > things safer, since the exports are enforced by the kernel. Without > this, a malicious NFSv4 client could "guess" a file handle for a file > outside the jail and gain access to that file. Put another way, without > a separate file system, there is no way to stop a malicious client from > finding files above the Root file handle. (Normal clients will use > PutRootFH and LookupParent and these won't be able to go above the top > of the jail.) > > So, what do others think of enforcing the requirement that each jail > have its own file systems for this? I don't care for any of it. It looks like additional overhead with the addition of potential security risks. All for a very limited (and as yet unknown) use case. I am thinking that if/when this goes into main, it would be under a new kernel build option called something like NFSD_VIMAGE. I think that would avoid the overhead/security risks for those that do not need/want it. Brilliant. Count me in. :-) --chris rick --chris > > rick > > >> - Peter >> >> >> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem wrote: >> >>> Hi, >>> >>> bz@ has encouraged me to fiddle with the nfsd >>> so that it works in a vnet jail. >>> I have now basically done so, specifically for >>> NFSv4, since NFSv3 presents various issues. >>> >>> What I have not yet done is put global variables >>> in the vnet. This needs to be done so that the nfsd >>> can be run in multiple jail instances and/or in and >>> outside of a jail. >>> The problem is that there are 100s of global variables. >>> >>> I can see two approaches: >>> 1 - Move them all into the vnet jail. This would imply >>> that all the sysctls need to somehow be changed, >>> which would seem to be a POLA violation. >>> It also implies a lot of stuff in the vnet. >>> 2 - Just move the global variables that will always >>> differ from one nfsd to another (this would make >>> the sysctls global and apply to all nfsds). >>> This will keep the number of globals in the vnet >>> smaller. >>> >>> I am currently leaning towards #2, put what do others >>> think? >>> >>> rick >>> ps: Personally, I don't know what use there is of >>> running the nfsd inside a vnet jail, but bz@ has >>> some use case. >>> >> >> 0xBDE49540.asc Description: application/pgp-keys
Re: RFC: nfsd in a vnet jail
On Thu, Dec 1, 2022 at 8:23 AM Chris wrote: > On 2022-11-29 16:21, Rick Macklem wrote: > > On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson > wrote: > > > >> Keep the global variables as defaults that apply to all nfsds and allow > >> (at least some subset) to be overridden inside the net jails if some > things > >> need to be changed from the defaults? > >> > >> This is pretty much a reply to one of the posts selected at random, > > but I thought that better than starting a new email thread. > > > > bz@ and asomers@ have both asked about running mountd within a vnet > prison > > (one via offlist email and the other on phabricator). > > > > I think it is worth discussing here... > > mountd (rightly or wrongly) does two distinctly different things: > > 1 - It pushes the exports into the kernel via nmount() so they > > can be hung off of the "struct mount" for a file system's > > mount point. > > --> This can only work for file system mount points and can > > only be done once for any given file system mount point. > > > > At this time, I have it done once globally outside of the prisons. > > The alternative I can see is doing it within each prison, but I > > think that would require that each prison have its own file > system(s). > > (ie. The prison's root would always be a file system mount point.) > > > > 2 - It handles RPC Mount protocol requests from NFSv3 clients. This one > > is NFSv3 specific, which is why I have done this NFSv4 only at > > this time. To do this, it must be able to register with rpcbind, > > and I have no idea if running rpcbind in a vnet jail is practical. > > > > Enforcing the use for separate file systems for each jail also makes > > things safer, since the exports are enforced by the kernel. Without > > this, a malicious NFSv4 client could "guess" a file handle for a file > > outside the jail and gain access to that file. Put another way, without > > a separate file system, there is no way to stop a malicious client from > > finding files above the Root file handle. (Normal clients will use > > PutRootFH and LookupParent and these won't be able to go above the top > > of the jail.) > > > > So, what do others think of enforcing the requirement that each jail > > have its own file systems for this? > > I don't care for any of it. It looks like additional overhead with the > addition of potential security risks. All for a very limited (and as yet > unknown) use case. > I am thinking that if/when this goes into main, it would be under a new kernel build option called something like NFSD_VIMAGE. I think that would avoid the overhead/security risks for those that do not need/want it. rick > > --chris > > > > rick > > > > > >> - Peter > >> > >> > >> On Fri, Nov 25, 2022, 4:24 PM Rick Macklem > wrote: > >> > >>> Hi, > >>> > >>> bz@ has encouraged me to fiddle with the nfsd > >>> so that it works in a vnet jail. > >>> I have now basically done so, specifically for > >>> NFSv4, since NFSv3 presents various issues. > >>> > >>> What I have not yet done is put global variables > >>> in the vnet. This needs to be done so that the nfsd > >>> can be run in multiple jail instances and/or in and > >>> outside of a jail. > >>> The problem is that there are 100s of global variables. > >>> > >>> I can see two approaches: > >>> 1 - Move them all into the vnet jail. This would imply > >>> that all the sysctls need to somehow be changed, > >>> which would seem to be a POLA violation. > >>> It also implies a lot of stuff in the vnet. > >>> 2 - Just move the global variables that will always > >>> differ from one nfsd to another (this would make > >>> the sysctls global and apply to all nfsds). > >>> This will keep the number of globals in the vnet > >>> smaller. > >>> > >>> I am currently leaning towards #2, put what do others > >>> think? > >>> > >>> rick > >>> ps: Personally, I don't know what use there is of > >>> running the nfsd inside a vnet jail, but bz@ has > >>> some use case. > >>> > >> > >> >
Re: RFC: nfsd in a vnet jail
On Thu, Dec 1, 2022 at 2:01 AM Milan Obuch wrote: > On Thu, 01 Dec 2022 10:29:25 +0100 > Alexander Leidinger wrote: > > > Quoting Alan Somers (from Tue, 29 Nov 2022 > > 17:28:10 -0700): > > > > > On Tue, Nov 29, 2022 at 5:21 PM Rick Macklem > > > wrote: > > > > >> So, what do others think of enforcing the requirement that each > > >> jail have its own file systems for this? > > > > > > I think that's a totally reasonable requirement. Especially so for > > > ZFS users, who already create a filesystem per jail for other > > > reasons. > > > > While I agree that it is a reasonable requirement, just a note that > > we can not assume that every existing jail resides on its own file > > system. The base system jail infrastructure doesn't check this, and > > the ezjail port doesn't either. The iocage port does it. > > > > My position would be 'recommended, but not forced-to' one. I have > various installations with jails sharing parts of filesystem (like > ports or src tree for development, or even local git repository), or > even running with exactly the same directory as root of number of > jails. Probably not a common scenario for sure, but still useful. > Others indicate they want mountd to run inside the jail. To get that to work, the jail needs to be in a separate file system, since it is the file system(s) mount point(s) that the export information is attached to in the kernel. It comes down to... #1 - Run mountd outside of the jails and encourage use of separate file systems. (Also, since the exports information would be applied to the file system(s) and not the jails, a malicious NFS client could "guess" a file handle and access files outside of the jail. This seems counter to what a jail should provide.) OR #2 - Require separate file systems and run mountd inside the jail(s). I think that allowing both alternatives would be too confusing and it seems that most want mountd to run within the jail(s). As such, unless others prefer #1, I think #2 is the way to go. rick > > Regards, > Milan >
Re: RFC: nfsd in a vnet jail
On Thu, Dec 1, 2022 at 1:29 AM Alexander Leidinger wrote: > > Quoting Alan Somers (from Tue, 29 Nov 2022 > 17:28:10 -0700): > > > On Tue, Nov 29, 2022 at 5:21 PM Rick Macklem > wrote: > > >> So, what do others think of enforcing the requirement that each jail > >> have its own file systems for this? > > > > I think that's a totally reasonable requirement. Especially so for > > ZFS users, who already create a filesystem per jail for other reasons. > > While I agree that it is a reasonable requirement, just a note that we > can not assume that every existing jail resides on its own file > system. The base system jail infrastructure doesn't check this, and > the ezjail port doesn't either. The iocage port does it. > > Is there a way to detect this inside a jail and error out in nfsd/mountd? I think the check (...->pr_root->v_vflag & VV_ROOT) is sufficient. At least it is working for current testing. rick > > Bye, > Alexander. > > -- > http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF > http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF >
Buildword Error on a recent -CURRENT
Hello, I am currently having the following build work error on a recent -CURRENT. = ===> usr.bin/less (all) -16372 bytes available --- loader_lua.bin --- *** [loader_lua.bin] Error code 1 make[5]: stopped in /boiler/nfs/src/stand/i386/loader_lua --- all_subdir_stand --- make[2]: stopped in /boiler/nfs/src --- all_subdir_lib --- make[2]: stopped in /boiler/nfs/src --- all_subdir_usr.bin --- make[2]: stopped in /boiler/nfs/src --- all_subdir_usr.sbin --- make[2]: stopped in /boiler/nfs/src 240.29 real 849.36 user73.97 sys --- everything --- make[1]: stopped in /boiler/nfs/src --- buildworld --- make: stopped in /boiler/nfs/src make[1]: "/boiler/nfs/src/Makefile.inc1" line 331: SYSTEM_COMPILER: libclang will be built for bootstrapping a cross-compiler . make[1]: "/boiler/nfs/src/Makefile.inc1" line 334: SYSTEM_LINKER: Determined that LD=ld matches the source tree. Not bootstr apping a cross-linker. Has anyone seen this error before or has recommandation on how to solve it. My src.conf for building -CURRENT is the following: WITH_EXTRA_TCP_STACKS=1 WITH_BEARSSL=1 WITH_PIE=1 WITH_RETPOLINE=1 WITH_INIT_ALL_ZERO=1 WITH_OPENSSL_KTLS=1 WITHOUT_CLEAN=1 The usual step if something with WITHOUT_CLEAN isn't working (deleting the obj directory) had no effect. --Gordon signature.asc Description: PGP signature
Re: RFC: nfsd in a vnet jail
On 2022-12-01 08:37, Alan Somers wrote: I don't care for any of it. It looks like additional overhead with the addition of potential security risks. All for a very limited (and as yet unknown) use case. Here's an example of a real-world use case. I'm responsible for supporting multiple products involving NFS, iSCSI, and other protocols. For security reasons, each product is placed on its own VLAN. Sometimes it's not practical to dedicate a physical server to a single product, so I have to double-up. For the products that don't involve NFS or iSCSI, I place them in a VNET jail. That way their processes can only access the correct VLAN. But NFS and iSCSI can't (yet) be jailed, so those products need to be served by JID 0. Therefore, those products' processes can access each other's VLANs. Clearly that's not ideal. Jailing different products is also good for manageability. It's easier to manage the list of packages that must be installed for each product, config file settings, etc. For example, some of our NFS products require vfs.nfsd.enable_stringtouid=1, but others could work without it. Right now, we're forced to turn it on for all products. OK. I can see that. Assuming I understand your intent correctly, I might choose bhyve(8) for that. Tho that *would* be a little more expensive. I rely on jail(8) daily. They're cheap, fast && easy. As yet, I've never had unreasonable concern for security. The proposed change looks like a potentially high addition of overhead (jail not so cheap now?). RPC && NFS are not cheap && have a comparatively high attack surface. So I guess my concerns/view are affected by this understanding. Thanks for the clarification, Alan. --chris -Alan 0xBDE49540.asc Description: application/pgp-keys
Re: RFC: nfsd in a vnet jail
> I don't care for any of it. It looks like additional overhead with the > addition of potential security risks. All for a very limited (and as yet > unknown) use case. Here's an example of a real-world use case. I'm responsible for supporting multiple products involving NFS, iSCSI, and other protocols. For security reasons, each product is placed on its own VLAN. Sometimes it's not practical to dedicate a physical server to a single product, so I have to double-up. For the products that don't involve NFS or iSCSI, I place them in a VNET jail. That way their processes can only access the correct VLAN. But NFS and iSCSI can't (yet) be jailed, so those products need to be served by JID 0. Therefore, those products' processes can access each other's VLANs. Clearly that's not ideal. Jailing different products is also good for manageability. It's easier to manage the list of packages that must be installed for each product, config file settings, etc. For example, some of our NFS products require vfs.nfsd.enable_stringtouid=1, but others could work without it. Right now, we're forced to turn it on for all products. -Alan
Re: RFC: nfsd in a vnet jail
On 2022-11-29 16:21, Rick Macklem wrote: On Sun, Nov 27, 2022 at 10:04 AM Peter Eriksson wrote: Keep the global variables as defaults that apply to all nfsds and allow (at least some subset) to be overridden inside the net jails if some things need to be changed from the defaults? This is pretty much a reply to one of the posts selected at random, but I thought that better than starting a new email thread. bz@ and asomers@ have both asked about running mountd within a vnet prison (one via offlist email and the other on phabricator). I think it is worth discussing here... mountd (rightly or wrongly) does two distinctly different things: 1 - It pushes the exports into the kernel via nmount() so they can be hung off of the "struct mount" for a file system's mount point. --> This can only work for file system mount points and can only be done once for any given file system mount point. At this time, I have it done once globally outside of the prisons. The alternative I can see is doing it within each prison, but I think that would require that each prison have its own file system(s). (ie. The prison's root would always be a file system mount point.) 2 - It handles RPC Mount protocol requests from NFSv3 clients. This one is NFSv3 specific, which is why I have done this NFSv4 only at this time. To do this, it must be able to register with rpcbind, and I have no idea if running rpcbind in a vnet jail is practical. Enforcing the use for separate file systems for each jail also makes things safer, since the exports are enforced by the kernel. Without this, a malicious NFSv4 client could "guess" a file handle for a file outside the jail and gain access to that file. Put another way, without a separate file system, there is no way to stop a malicious client from finding files above the Root file handle. (Normal clients will use PutRootFH and LookupParent and these won't be able to go above the top of the jail.) So, what do others think of enforcing the requirement that each jail have its own file systems for this? I don't care for any of it. It looks like additional overhead with the addition of potential security risks. All for a very limited (and as yet unknown) use case. --chris rick - Peter On Fri, Nov 25, 2022, 4:24 PM Rick Macklem wrote: Hi, bz@ has encouraged me to fiddle with the nfsd so that it works in a vnet jail. I have now basically done so, specifically for NFSv4, since NFSv3 presents various issues. What I have not yet done is put global variables in the vnet. This needs to be done so that the nfsd can be run in multiple jail instances and/or in and outside of a jail. The problem is that there are 100s of global variables. I can see two approaches: 1 - Move them all into the vnet jail. This would imply that all the sysctls need to somehow be changed, which would seem to be a POLA violation. It also implies a lot of stuff in the vnet. 2 - Just move the global variables that will always differ from one nfsd to another (this would make the sysctls global and apply to all nfsds). This will keep the number of globals in the vnet smaller. I am currently leaning towards #2, put what do others think? rick ps: Personally, I don't know what use there is of running the nfsd inside a vnet jail, but bz@ has some use case. 0xBDE49540.asc Description: application/pgp-keys
Re: RFC: nfsd in a vnet jail
On Thu, Dec 1, 2022 at 2:30 AM Alexander Leidinger wrote: > > Quoting Alan Somers (from Tue, 29 Nov 2022 > 17:28:10 -0700): > > > On Tue, Nov 29, 2022 at 5:21 PM Rick Macklem > wrote: > > >> So, what do others think of enforcing the requirement that each jail > >> have its own file systems for this? > > > > I think that's a totally reasonable requirement. Especially so for > > ZFS users, who already create a filesystem per jail for other reasons. > > While I agree that it is a reasonable requirement, just a note that we > can not assume that every existing jail resides on its own file > system. The base system jail infrastructure doesn't check this, and > the ezjail port doesn't either. The iocage port does it. > I have several jails that all live on the same zfs data set that I setup ages ago before I understood the full benefits of ZFS... but I could migrate in a pinch. But they aren't in their own vnet, so maybe that doesn't apply. > Is there a way to detect this inside a jail and error out in nfsd/mountd? > Whatever we do, there will be people bitten by it, so we need to make the messaging around it good (the error messages from the system, as well as the documentation). Warner
Re: RFC: nfsd in a vnet jail
On Thu, 01 Dec 2022 10:29:25 +0100 Alexander Leidinger wrote: > Quoting Alan Somers (from Tue, 29 Nov 2022 > 17:28:10 -0700): > > > On Tue, Nov 29, 2022 at 5:21 PM Rick Macklem > > wrote: > > >> So, what do others think of enforcing the requirement that each > >> jail have its own file systems for this? > > > > I think that's a totally reasonable requirement. Especially so for > > ZFS users, who already create a filesystem per jail for other > > reasons. > > While I agree that it is a reasonable requirement, just a note that > we can not assume that every existing jail resides on its own file > system. The base system jail infrastructure doesn't check this, and > the ezjail port doesn't either. The iocage port does it. > My position would be 'recommended, but not forced-to' one. I have various installations with jails sharing parts of filesystem (like ports or src tree for development, or even local git repository), or even running with exactly the same directory as root of number of jails. Probably not a common scenario for sure, but still useful. Regards, Milan
Re: RFC: nfsd in a vnet jail
Quoting Alan Somers (from Tue, 29 Nov 2022 17:28:10 -0700): On Tue, Nov 29, 2022 at 5:21 PM Rick Macklem wrote: So, what do others think of enforcing the requirement that each jail have its own file systems for this? I think that's a totally reasonable requirement. Especially so for ZFS users, who already create a filesystem per jail for other reasons. While I agree that it is a reasonable requirement, just a note that we can not assume that every existing jail resides on its own file system. The base system jail infrastructure doesn't check this, and the ezjail port doesn't either. The iocage port does it. Is there a way to detect this inside a jail and error out in nfsd/mountd? Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF pgpRjJWWhBIKb.pgp Description: Digitale PGP-Signatur
