[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Following rules present in jaunty: KERNEL==umad[0-9]*, NAME=infiniband/%k KERNEL==issm[0-9]*, NAME=infiniband/%k KERNEL==ucm[0-9]*, NAME=infiniband/%k, MODE=0666 KERNEL==uverbs[0-9]*, NAME=infiniband/%k, MODE=0666 KERNEL==uat, NAME=infiniband/%k, MODE=0666 KERNEL==ucma, NAME=infiniband/%k, MODE=0666 KERNEL==rdma_cm, NAME=infiniband/%k, MODE=0666 ** Changed in: udev (Ubuntu) Status: New = Fix Released -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
We seem to be at consensus here, so I'll have the following rule merged upstream: KERNEL==rdma_cm, MODE=0666 ** Changed in: udev (Ubuntu) Sourcepackagename: librdmacm = udev Assignee: (unassigned) = Scott James Remnant (scott) -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
On Fri, 2009-01-23 at 05:46 +, Roland Dreier wrote: I missed a key part of this paragraph before. You say that the whole point is that unprivileged userspace applications can use RDMA directly? Yes, non-suid executables run by normal users should be able to use RDMA directly in a safe fashion. If that's the case, should these devices not simply have -rw-rw-rw permissions (like /dev/net/tun, /dev/fuse, etc.) so that all userspace applications can use them? Having 0666 permissions would not necessarily be a bad idea, but the consensus among other distributions is to limit RDMA access to an rdma group so that administrators have some control over who gets direct hardware access Any rule we add will be in upstream udev; so all the distributions would end up with it anyway. Upstream udev strongly discourages groups for device access that users are placed in. (even though in theory it is safe for anyone, there is the possibility of untrusted users consuming network bandwidth at least). It's pretty easy to consume network bandwidth from userspace, you open lots of sockets to somewhere and start reading or writing ;-) Likewise it's pretty trivial to consume memory. Also, RDMA often requires increasing the amount of locked memory allowed in /etc/security/limits.conf, and doing that by group rdma is convenient as well. So it sounds like there's other limits in place anyway to what people can do with RDMA? Sounds safe Given that you seem to have moved fuse from 0660 to 0666 between Intrepid and Jaunty, I guess it would be consistent to have the same permission for rdma access. Is there some reason that you keep the fuse group around and make /dev/fuse owned by it, or is that just a leftover from the old udev rules? The group is leftover from before. Scott -- Scott James Remnant sc...@canonical.com -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
I missed a key part of this paragraph before. You say that the whole point is that unprivileged userspace applications can use RDMA directly? Yes, non-suid executables run by normal users should be able to use RDMA directly in a safe fashion. If that's the case, should these devices not simply have -rw-rw-rw permissions (like /dev/net/tun, /dev/fuse, etc.) so that all userspace applications can use them? Having 0666 permissions would not necessarily be a bad idea, but the consensus among other distributions is to limit RDMA access to an rdma group so that administrators have some control over who gets direct hardware access (even though in theory it is safe for anyone, there is the possibility of untrusted users consuming network bandwidth at least). Also, RDMA often requires increasing the amount of locked memory allowed in /etc/security/limits.conf, and doing that by group rdma is convenient as well. Given that you seem to have moved fuse from 0660 to 0666 between Intrepid and Jaunty, I guess it would be consistent to have the same permission for rdma access. Is there some reason that you keep the fuse group around and make /dev/fuse owned by it, or is that just a leftover from the old udev rules? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Just winding back this discussion briefly: D-Bus/PolicyKit seems very much overengineered and too complex for this issue, and it doesn't fit the model of RDMA very well anyway, since the whole point of RDMA is that unprivileged userspace applications use RDMA hardware directly without the overhead of a system call into the kernel, let alone a D-Bus method call to another process. I missed a key part of this paragraph before. You say that the whole point is that unprivileged userspace applications can use RDMA directly? If that's the case, should these devices not simply have -rw-rw-rw permissions (like /dev/net/tun, /dev/fuse, etc.) so that all userspace applications can use them? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
I'm unsubscribing u-u-s for now waiting for a debdiff against current Jaunty version. Could you please coordinate with Scott to have this uploaded? Thanks! -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
On Wed, 2008-11-19 at 16:22 +, Roland Dreier wrote: D-Bus/PolicyKit seems very much overengineered and too complex for this issue, and it doesn't fit the model of RDMA very well anyway, since the whole point of RDMA is that unprivileged userspace applications use RDMA hardware directly without the overhead of a system call into the kernel, let alone a D-Bus method call to another process. I don't agree. Adding a PolicyKit authorization to use RDMA devices is not practically any harder than adding a group; in fact, maintenance-wise it's substantially easier. HAL may then be used to apply an ACL to the devices automatically if you want raw library aaccess. Scott -- Scott James Remnant [EMAIL PROTECTED] -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Adding a PolicyKit authorization to use RDMA devices is not practically any harder than adding a group; in fact, maintenance-wise it's substantially easier. I have to admit I have no idea how to do that. But anyway, what's the point? No one is wants to change their app to talk to PolicyKit to request access to a device node just to run on Ubuntu. HAL may then be used to apply an ACL to the devices automatically if you want raw library aaccess. How would that work? On a system with the rdma_ucm module loaded (so /dev/infiniband/rdma_cm exists), I see nothing promising-looking in lshal to tell hal about. And what's the advantage to using hal to change group permissions on a file when udev can create it with the correct permissions? A very common use case would be a multi-user cluster, where MPI jobs spawn processes on many nodes using ssh, with possibly multiple different users processes running on a node at once. For example, Open MPI (already in the Ubuntu archive) is typically used with RDMA in this way. And MPI is hard enough to configure without trying to explain to users that they get permission denied errors because their ConsoleKit can't connect to the session bus or something like that. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
On Mon, 2008-11-17 at 18:23 +, Roland Dreier wrote: We are attempting to avoid groups for device node access wherever possible. I think I've gotten that message. But could you give a hint as to what alternative mechanism you are using? It's very frustrating to get replies to all my comments except for when I ask what you want me to do. Access to system devices is provided through the HAL or DeviceKit interface. Permission to access is managed through the PolicyKit layer, where the D-Bus system bus service providing the device access negotiates privilege with the application requesting it. Scott -- Scott James Remnant [EMAIL PROTECTED] -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
D-Bus/PolicyKit seems very much overengineered and too complex for this issue, and it doesn't fit the model of RDMA very well anyway, since the whole point of RDMA is that unprivileged userspace applications use RDMA hardware directly without the overhead of a system call into the kernel, let alone a D-Bus method call to another process. Anyway I don't think it makes sense to try and implement what you're talking about just for Ubuntu, so I suggest you close this bug as won't fix and I'll just point people at fixed packages in my PPA. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
On Sun, 2008-11-16 at 05:17 +, Roland Dreier wrote: Is now an appropriate time to address this for Jaunty? We can have the discussion, This is very low down my priority stack right now - with a resync with the upstream udev rules much higher on it. Scott -- Scott James Remnant [EMAIL PROTECTED] -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Yes, let's have the discussion please. I understand that this isn't a high priority for you, but taking a little time early in the release cycle seems to be the only way to get this resolved for Jaunty. I don't see any way to make progress unless you give a hint as to what you feel is a better solution than my proposed patch. We should also take into account the fact that the libibverbs package has shipped group rdma rules for other device nodes for at least two Ubuntu releases now. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Past history is irrelevant. We are attempting to avoid groups for device node access wherever possible. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Past history is irrelevant. I think backwards compatibility and simplicity count for something. The only point about libibverbs is that the group rdma permissions are already applied to /dev/infiniband/uverbsX, and the class of users/applications that use those nodes are the same as the ones that would use /dev/infiniband/rdma_cm. So given that any systems where libibverbs is in use have already configured the rdma group, it seems we should at least consider whether we want to break that setup and/or introduce a different mechanism to do the same thing. We are attempting to avoid groups for device node access wherever possible. I think I've gotten that message. But could you give a hint as to what alternative mechanism you are using? It's very frustrating to get replies to all my comments except for when I ask what you want me to do. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Is now an appropriate time to address this for Jaunty? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
On Fri, 2008-10-24 at 02:03 +, Roland Dreier wrote: Seems like time is running out to address this in 8.10? Sorry, I mean 9.04 Scott -- Scott James Remnant [EMAIL PROTECTED] -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Seems like time is running out to address this in 8.10? This isn't really a security issue -- the rdma_cm device node is designed to be safe for unprivileged users to access, and Debian has been shipping udev rules that give group rdma access for quite a while with no reported security issues. And given that no users are in group rdma by default anyway, administrator intervention is required for this to make a difference even with the patch applied. I thought the objection was to having an rdma group, and I'd like to make progress on some more acceptable alternative mechanism, but I need some hint as to what that would be. -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
How do we make progress on this? As it stands only root (or suid apps) can use librdmacm with Intrepid. (Intrepid is the first release to include librdmacm) -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
We need to think about it. Since this is an issue of security and permissions, it's far better to ship requiring root access or administrator intervention than it is to ship with too light permissions. This will be addressed in 8.10 -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
While this is technically correct, by policy, I would hesitate before applying this to Intrepid. Too often, upstreams shirk the entire permissions problem by just telling distributions to create another group and put users into it. It's neither a scalable nor even desirable solution, because it isn't a solution - it's just a workaround. Instead we should ask more fundamental questions. What is RDMA? What kind of user would need access to these device nodes? How will they use them? Are they connected to some kind of physical hardware attached to the machine, or a pluggable device that any user who inserts it at their seat would expect to be able to use? Do users ever use these devices directly, or do they run programs that access them by talking a special protocol? Do users even run these programs at all, or are they daemons that manage the device, and present a user-space interface of their own? If a device is present on the system, should any user be able to access it? Or is it a privilege only for certain users, or even the system adminstrator? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Fair questions, although it probably would have been better to look at this before the other group rdma changes went into libibverbs as part of hardy (see bug #225788). And I wish we could have had this discussion two months ago, rather than two weeks before the Intrepid release. Anyway, I'll try to answer the questions: - RDMA stands for remote direct memory access, and it is a type of high performance networking implemented by InfiniBand and some 10 GbE adapters. Part of RDMA is kernel bypass, which allows userspace process direct access to hardware registers to reduce latency and CPU overhead in performing RDMA operations. http://en.wikipedia.org/wiki/RDMA has a more complete overview. - Users that are running high-performance jobs would need access to these device nodes; it makes sense to me that administrators would not necessarily want to allow all users to have direct access to do things that might interfere with other jobs on a high-performance network. - The device nodes in this particular bug are actually virtual devices that are used for connection setup; the actual direct-access nodes have permissions covered by the udev rules in the libibverbs1 package. In any case, RDMA hardware is generally a PCI Express or PCI-X card (basically a high-end NIC), although some systems have hardware directly on a system bus (AMD hypertransport, or IBM system p GX bus). The hardware is only pluggable via something like PCI hot-swap, which is generally a high-end server feature. It's definitely not something that a user on a multi-seat system is going to plug into a USB port. - Not sure what it would mean for users to use the devices directly -- obviously device access is through software (rather than poking solder balls with a wire or something like that). For the rdma_cm node specifically that this bug is about, typical user will link their application to librdmacm and use the library to establish RDMA connections. Users will then run their application directly (or possibly through a job submission queue for large shared clusters). - As I said before, the rdma_cm device nodes should be usable by non- administrator users, but the administrator probably wants the ability to restrict access to only certain users. Let me ask on fundamental question of my own: if upstreams are shirking responsibility by suggesting that standard group permissions be used by administrators to set policy, what do you feel is a better way for upstreams to provide this mechanism? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Is there any possibility of this getting reviewed and sponsored for Intrepid? -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 256216] Re: Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule
Here is a debdiff that bumps the version to -ubuntu1 and adds a librdmacm1.udev file with the required udev rule. It would be great if this could be integrated into the Ubuntu package ** Attachment added: librdmacm-udev.patch http://launchpadlibrarian.net/16665354/librdmacm-udev.patch -- Ubuntu is missing /dev/infiniband/rdma_cm group ownership udev rule https://bugs.launchpad.net/bugs/256216 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
