Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 2016-11-01 14:10:44 -0400 (-0400), Ben Swartzlander wrote: > On 11/01/2016 12:03 PM, Jeremy Stanley wrote: [...] > > into it the kernel's process space. This is the closest analogue we > > have to our drivers importing proprietary Python modules. > > I'm not talking about python code here. I'm talking about non-python > binaries, which were a large part of the discussion at the summit. [...] Thanks--from Sean's summary and some of the in-person recounting I've gotten from others who attended the session, it seemed like there might have been some fuzziness on that point. This could simply be due to terminology mismatches or lack of specificity. > I think we're saying the same thing. Python code can and should be 100% > Apache licensed. External C/C++/Java tools required to interact with a > storage controller could be treated similarly to Linux firmware and allowed > only if those tools could be made available under a freely distributable > license. From the conversations I've had I believe this would address the > major sticking point with Sean's "Option 3". Yes, I concur. I suggest however that we should make it clear this is not the preferred implementation model for drivers, and then supply guidance/recommendations on alternative designs that result in a completely open driver for those willing and able to provide one. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 11/01/2016 12:03 PM, Jeremy Stanley wrote: On 2016-11-01 00:26:11 -0400 (-0400), Ben Swartzlander wrote: [...] As usual I'd like to point to the Linux project as a good example for how to handle such things. Linux is older than us and has been dealing with drivers and proprietary code for a very long time. Linux does a few specific things that I like a lot: [...] 3) Drivers which require proprietary stuff (typically called "firmware" or "binary blobs" in the Linux world) can contribute that stuff to the linux-firmware repo which supports closed-source but freely-distributable software. Note that the "proprietary stuff" here is generally opaque blobs the running kernel uploads into other parts of the system to run on or initialize different processors than the kernel itself is running within. I'm pretty sure the Linux devs wouldn't (and legally couldn't? but I am not a lawyer nor a kernel maintainer) allow drivers in tree that dynamically load proprietary external libraries into it the kernel's process space. This is the closest analogue we have to our drivers importing proprietary Python modules. I'm not talking about python code here. I'm talking about non-python binaries, which were a large part of the discussion at the summit. Personally I think the needs of the distros could be met with something like the linux-firmware repo. It would create a way for vendors that require proprietary stuff to make it available for distros to do what they need to do. [...] This might be a solution to the case where drivers call proprietary local command-line utilities or load proprietary data for use in initializing some device, as long as those things are still freely redistributable on their own. I don't know, though, how many services have drivers in exactly this situation. It certainly doesn't change the legality of things like importing proprietary Python modules within drivers, and is also a suboptimal solution I think we should discourage in favor of providing _actual_ free drivers. I suppose it's worth a try, but I'm skeptical it will significantly alter the dynamics of the situation. I think we're saying the same thing. Python code can and should be 100% Apache licensed. External C/C++/Java tools required to interact with a storage controller could be treated similarly to Linux firmware and allowed only if those tools could be made available under a freely distributable license. From the conversations I've had I believe this would address the major sticking point with Sean's "Option 3". -Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 2016-11-01 12:29:27 -0500 (-0500), Sean McGinnis wrote: [...] > Is importing proprietary libs really a legal issue? If it happens at > runtime based on how the user configured the system, and they have > whatever licensing rights _they_ need to use that library/CLI, I > don't see an issue with that. As I understand it (not a lawyer!) the legal issue is on our end distributing a driver that is designed to import and make calls into some library which has incompatible licensing (even if we don't redistribute that library), as it becomes a derivative work of that library and is therefore contaminated by its license. There are those who disagree with this copyright position (and who might similarly disagree that an linking a non-GPL-compatible library in a GPL'd C application is similarly fine to distribute), but I would be surprised if we as a community could get consensus that it's a legally safe position to take on the issue. Further, we are pretty explicit in our technical governance documents as to what licenses are acceptable for dependencies of OpenStack services, and to a great extent this is informed by whether it's _legal_ for us or others to distribute software that imports and calls into libraries even if we don't "depend" on them in a strict sense. Command line interfaces are for the most part, I gather, not subject to the same issues (for reasons that frankly seem a little absurd to me). I'm going to refrain from citing sources here because we're already wandering into territory that's better handled on the legal-discuss@ ML. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On Tue, Nov 01, 2016 at 03:47:11PM +, Jeremy Stanley wrote: > On 2016-10-31 13:23:48 -0500 (-0500), Sean McGinnis wrote: > > Last Tuesday we had a cross-project session around the use of > > proprietary code/libs/binaries in OpenStack drivers. The challenge > > we are running into is where to draw the line with these drivers. > > What is the appropriate policy to have in place to allow/disallow > > certain approaches. > [...] > > Thanks for summarizing this! I desperately wanted to attend but > ended up with other obligations for much of Tuesday and so could > not. > > [some comments follow which could be confused as legal advice, so > insert obligatory "I am not a lawyer" boilerplate here] > > > Option 1: > > > > - All libraries imported by the driver must be licensed such that > > they are redistributable by package maintainers. > > Which means not just licensed for redistribution, but released under > an Apache-compatible license (for example, the FSF does not consider > the Apache License v2 combatible with GPLv2 so a driver library > under the latter license would likely present a legal problem). See > http://governance.openstack.org/reference/licensing.html for more. > Also, I don't think this bullet is unique to option 1. We have an > absolute need for software we produce to be legally redistributable, > and if it actively imports (in the Python sense, or links against in > the C sense) proprietary or otherwise incompatibly-licensed > libraries we can't legally distribute what we're writing because it > (according to general opinion on software copyright) would be a > derivative work of software under that other license. > > > - Existing non-compliant driver code would need to be updated by > > the Q release to stay in tree. > > - Code that does not get imported into the driver at runtime > > (CLIs, external binaries, remote application servers) are > > acceptable to be not redistributable. > > I'd love if we could take it a step further and _recommend_ against > drivers calling proprietary local tools, even if we state it's > strictly acceptable to do so. Some vendors go this route because > they have a strong preference for the model (e.g. they already have > a non-free CLI tool for this and want to avoid costly code > duplication), but others may be doing it simply because they're > copying that pattern from an existing driver without realizing it's > adding an unnecessary burden on users, operators and deployers. The > idealist in me shudders at the idea of calling it acceptable, but my > inner pragmatist says we should at least provide some guidance > around this choice in favor of the open freeness valued by our > community. > > > Issues: > > > > - The desire for having things redistributable was to have > > everything fully functional "out of the box" no matter how the > > deployer configured the system. > > Having not been there, I can't say for sure but this sounds like a > mischaracterization of the position. At least for me, the desire for > having things redistributable is so that downstream consumers of > OpenStack are not beholden to the vendors just to be able to use our > (free!) software with hardware they have: > > As a thought exercise, say I have a Gralpy brand Frobnule v3 widget > in my environment and want to use it with OpenStack Plugh. The > Frobnule driver for Plugh calls a proprietary `frobnitz` command > line tool to do everything, but that tool has stopped supporting > FrobOS work with my hardware. Or maybe I'm annoyed by GralpyCo's > extortionist support contracts which are required to be able to > download that tool. Or maybe GralpyCo has folded entirely and their > software site is now completely offline. > > While you might retort that operators should avoid using devices > which are EOL, for which they can't afford a support contract, or > whose vendors have gone out of business, there are lots of reasons > why you might. I personally have lots of old hardware, the > manufacturers of whom have not existed for years, but because the > drivers for it are actually implemented as free software rather than > openwashing proprietary tools with a "free" driver library I'm able > to use them with the latest versions of my free software of choice. > > Another _very_ good reason to discourage drivers implemented as > opaque proprietary utilities is that it severely complicates > reverse-engineering of the actual interfaces being called, so in the > Frobnule case above if GralpyCo doesn't want to support some feature > of the hardware in the `frobnitz` utility then its nontrivial for > someone else (say a non-Gralpy employee) in our community to extend > the driver in OpenStack Plugh to do so. > > > I don't believe this is ever possible. With requirements of > > setting up some type of application or management server to > > enable some solutions, it is not possible to package all > > required code and binaries for every configuration. >
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On Tue, Nov 01, 2016 at 12:26:11AM -0400, Ben Swartzlander wrote: > On 10/31/2016 02:23 PM, Sean McGinnis wrote: > >Last Tuesday we had a cross-project session around the use of proprietary > >code/libs/binaries in OpenStack drivers. The challenge we are running into > >is where to draw the line with these drivers. What is the appropriate policy > >to have in place to allow/disallow certain approaches. > > > > ..snip.. > > > > As usual I'd like to point to the Linux project as a good example for how to > handle such things. Linux is older than us and has been dealing with drivers > and proprietary code for a very long time. > > Linux does a few specific things that I like a lot: > 1) Drivers have to match the license (GPLv2 in Linux's case) which > guarantees the code is free software. > 2) Drivers have to be in tree, which makes it easier for kernel developers > to evolve the driver interfaces without breaking most drivers. > 3) Drivers which require proprietary stuff (typically called "firmware" or > "binary blobs" in the Linux world) can contribute that stuff to the > linux-firmware repo which supports closed-source but freely-distributable > software. I do think it's good to look at what is done with Linux as a well established example. But I also think it's a little different, so we can't really compare apples to apples here. Linux is now a very mature installable OS that is well supported with vendors devoting resources to making sure their hardware works well with it. OpenStack is a configurable platform. Some assembly required, even if you go with all of the defaults. I think we're closer to the earlier days of Linux. I remember spending a lot of time trying to get the right source directly from a Toshiba site to be able to compile in support for my laptop's NIC to work with Red Hat (3?). In the case of Linux, this was just early days and vendors were just catching on. (Really, why would anyone want to run Linux on a laptop?? That's for servers!) But in our case I don't think it's just a matter of maturity. There will always be proprietary components that need to be installed and configured for different options. > > Personally I think the needs of the distros could be met with something like > the linux-firmware repo. It would create a way for vendors that require > proprietary stuff to make it available for distros to do what they need to > do. > > Regarding drivers that "cheat" and put all the important logic outside the > open source driver code, I think we can safely ignore these. These drivers > hurt nothing but themselves by hiding their logic. The chance of breaking > them with a change to the driver interface is higher but this is a GOOD > thing -- it discourages what we perceive to be bad behavior. The Linux > community has already fought this battle and while some companies continue > to supply out of tree drivers, the process is typically very painful for > those maintainers and it motivates them to eventually open source their > drivers and put them in the tree. > > -Ben > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 2016-11-01 00:26:11 -0400 (-0400), Ben Swartzlander wrote: [...] > As usual I'd like to point to the Linux project as a good example > for how to handle such things. Linux is older than us and has been > dealing with drivers and proprietary code for a very long time. > > Linux does a few specific things that I like a lot: [...] > 3) Drivers which require proprietary stuff (typically called > "firmware" or "binary blobs" in the Linux world) can contribute > that stuff to the linux-firmware repo which supports closed-source > but freely-distributable software. Note that the "proprietary stuff" here is generally opaque blobs the running kernel uploads into other parts of the system to run on or initialize different processors than the kernel itself is running within. I'm pretty sure the Linux devs wouldn't (and legally couldn't? but I am not a lawyer nor a kernel maintainer) allow drivers in tree that dynamically load proprietary external libraries into it the kernel's process space. This is the closest analogue we have to our drivers importing proprietary Python modules. > Personally I think the needs of the distros could be met with > something like the linux-firmware repo. It would create a way for > vendors that require proprietary stuff to make it available for > distros to do what they need to do. [...] This might be a solution to the case where drivers call proprietary local command-line utilities or load proprietary data for use in initializing some device, as long as those things are still freely redistributable on their own. I don't know, though, how many services have drivers in exactly this situation. It certainly doesn't change the legality of things like importing proprietary Python modules within drivers, and is also a suboptimal solution I think we should discourage in favor of providing _actual_ free drivers. I suppose it's worth a try, but I'm skeptical it will significantly alter the dynamics of the situation. -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 2016-10-31 13:23:48 -0500 (-0500), Sean McGinnis wrote: > Last Tuesday we had a cross-project session around the use of > proprietary code/libs/binaries in OpenStack drivers. The challenge > we are running into is where to draw the line with these drivers. > What is the appropriate policy to have in place to allow/disallow > certain approaches. [...] Thanks for summarizing this! I desperately wanted to attend but ended up with other obligations for much of Tuesday and so could not. [some comments follow which could be confused as legal advice, so insert obligatory "I am not a lawyer" boilerplate here] > Option 1: > > - All libraries imported by the driver must be licensed such that > they are redistributable by package maintainers. Which means not just licensed for redistribution, but released under an Apache-compatible license (for example, the FSF does not consider the Apache License v2 combatible with GPLv2 so a driver library under the latter license would likely present a legal problem). See http://governance.openstack.org/reference/licensing.html for more. Also, I don't think this bullet is unique to option 1. We have an absolute need for software we produce to be legally redistributable, and if it actively imports (in the Python sense, or links against in the C sense) proprietary or otherwise incompatibly-licensed libraries we can't legally distribute what we're writing because it (according to general opinion on software copyright) would be a derivative work of software under that other license. > - Existing non-compliant driver code would need to be updated by > the Q release to stay in tree. > - Code that does not get imported into the driver at runtime > (CLIs, external binaries, remote application servers) are > acceptable to be not redistributable. I'd love if we could take it a step further and _recommend_ against drivers calling proprietary local tools, even if we state it's strictly acceptable to do so. Some vendors go this route because they have a strong preference for the model (e.g. they already have a non-free CLI tool for this and want to avoid costly code duplication), but others may be doing it simply because they're copying that pattern from an existing driver without realizing it's adding an unnecessary burden on users, operators and deployers. The idealist in me shudders at the idea of calling it acceptable, but my inner pragmatist says we should at least provide some guidance around this choice in favor of the open freeness valued by our community. > Issues: > > - The desire for having things redistributable was to have > everything fully functional "out of the box" no matter how the > deployer configured the system. Having not been there, I can't say for sure but this sounds like a mischaracterization of the position. At least for me, the desire for having things redistributable is so that downstream consumers of OpenStack are not beholden to the vendors just to be able to use our (free!) software with hardware they have: As a thought exercise, say I have a Gralpy brand Frobnule v3 widget in my environment and want to use it with OpenStack Plugh. The Frobnule driver for Plugh calls a proprietary `frobnitz` command line tool to do everything, but that tool has stopped supporting FrobOSI don't believe this is ever possible. With requirements of > setting up some type of application or management server to > enable some solutions, it is not possible to package all > required code and binaries for every configuration. I wish you'd refrained from editorializing within your summary, and instead stated your personal beliefs in a followup reply to yourself or something. It makes it harder to untangle your personal biases from your retelling of the opinions of others. > Option 2: > > - Remove all drivers that are not completely open source and contained in the > project repo. > > Issues: > > - This restricts things down to only the drivers that can currently be run in > gate. > - This would be a big issue for many Cinder drivers and I'm sure most Ironic > drivers. > - I strongly believe this would result in less vendor involvement in upstream > work. I agree here. I'd prefer if it could be free all the way from an OpenStack service to a standard protocol interface (network communication, USB, PCI, DMA, whatever) but in my opinion the only thing worse for operators than needing to get software from all over the Internet to get their hardware working is to provide no support for their hardware at all because the vendor has thrown its hands up at our hard-line stance on software freedom. It's a delicate balance, surely. > Option 3: > > - Require the majority of business logic is in the open source code. This seems like it would be very hard to police in practice. How easy do we expect it will be to untangle "business logic" from other implementation? And how much of a "majority" will be "required?" > - Allow third party,
Re: [openstack-dev] [All] Cross project prorietary driver code recap
On 10/31/2016 02:23 PM, Sean McGinnis wrote: Last Tuesday we had a cross-project session around the use of proprietary code/libs/binaries in OpenStack drivers. The challenge we are running into is where to draw the line with these drivers. What is the appropriate policy to have in place to allow/disallow certain approaches. There was a lot of good discussion and input from multiple projects that are affected by this. I won't attempt to recap the full conversation here. The etherpad with the notes from the discussion can be found here: https://etherpad.openstack.org/p/ocata-xp-proprietary-drivers The two main concerns I heard were around 1) the ability to package and redistribute everything needed to run an OpenStack environment, and 2) the value these drivers bring to the OpenStack project. Particularly those that have no business logic in the code in OpenStack and just call out to third party libraries to handle all business logic. Proposals = Option 1: - All libraries imported by the driver must be licensed such that they are redistributable by package maintainers. - Existing non-compliant driver code would need to be updated by the Q release to stay in tree. - Code that does not get imported into the driver at runtime (CLIs, external binaries, remote application servers) are acceptable to be not redistributable. Issues: - The desire for having things redistributable was to have everything fully functional "out of the box" no matter how the deployer configured the system. I don't believe this is ever possible. With requirements of setting up some type of application or management server to enable some solutions, it is not possible to package all required code and binaries for every configuration. Option 2: - Remove all drivers that are not completely open source and contained in the project repo. Issues: - This restricts things down to only the drivers that can currently be run in gate. - This would be a big issue for many Cinder drivers and I'm sure most Ironic drivers. - I strongly believe this would result in less vendor involvement in upstream work. Option 3: - Require the majority of business logic is in the open source code. - Allow third party, non-redistributable libraries and CLIs that are used as more of an "RPC" type interface. - Reviewers should be able to review the driver code and at least get some idea of the steps the driver is doing to perform each requested operation. Issues: - Does not address the desire to package all needed requirements. My preference is actually option 3. The reality is, with most vendor solutions there's always going to be some part of the overall solution that is proprietary and requires some configuration and setup by the deployer. Arbitrarily drawing that line such that a proprietary app server is OK, but a proprietary library is not OK, just leads to poor solution architectures such as requiring the end user to set up a separate server just so the driver can SSH into it to run commands. The case that instigated this discussion for me was a proposal to have a driver that did nothing more than make a one line call of everything out to a library that handled all logic. In that case, there is no benefit to the community of being able to review the code and give some assurance that it is doing things correctly. And the only benefit really it gives the vendor is they have an out of tree driver that gets advertised as being in tree. My desire as a code reviewer and project maintainer is that I can take a look at a driver and at least get an idea of what they are doing and how their device works to perform some of these operations. That's not an exact number - I'm not saying something like 10% of the code needs to be in the driver, but 9% would not be acceptable - but we need to have some visiblity to the logic so we as a community can at least help point users in the right direction if they come looking for help. I think this works for Cinder and probably is necessary for Ironic. I'd be interested in hearing feedback or other options from other folks. I'd especially like to hear from Helion, OSP, Mirantis, and others that are doing packaging and support for these deployments to hear how this impacts the way they do things. As usual I'd like to point to the Linux project as a good example for how to handle such things. Linux is older than us and has been dealing with drivers and proprietary code for a very long time. Linux does a few specific things that I like a lot: 1) Drivers have to match the license (GPLv2 in Linux's case) which guarantees the code is free software. 2) Drivers have to be in tree, which makes it easier for kernel developers to evolve the driver interfaces without breaking most drivers. 3) Drivers which require proprietary stuff (typically called "firmware" or "binary blobs" in the Linux world) can contribute that stuff to the linux-firmware repo which supports closed-source
