Re: [openstack-dev] [neutron] requirements.txt: explicit vs. implicit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 14/08/14 18:33, Ben Nemec wrote: On 08/14/2014 08:37 AM, Ihar Hrachyshka wrote: Hi all, some plugins depend on modules that are not mentioned in requirements.txt. Among them, Cisco Nexus (ncclient), Brocade (ncclient), Embrane (heleosapi)... Some other plugins put their dependencies in requirements.txt though (like Arista depending on jsonrpclib). There are pros and cons in both cases. The obvious issue with not putting those requirements in the file is that packagers are left uninformed about those implicit requirements existing, meaning plugins are shipped to users with broken dependencies. It also means we ship code that depends on unknown modules grabbed from random places in the internet instead of relying on what's available on pypi, which is a bit scary. With my packager hat on, I would like to suggest to make those dependencies explicit by filling in requirements.txt. This will make packaging a bit easier. Of course, runtime dependencies being set correctly do not mean plugins are working and tested, but at least we give them chance to be tested and used. But, maybe there are valid concerns against doing so. In that case, I would be glad to know how packagers are expected to track those implicit dependencies. I would like to ask community to decide what's the right way to handle those cases. So I raised a similar issue about six months ago and completely failed to follow up on the direction everyone seemed to be onboard with: http://lists.openstack.org/pipermail/openstack-dev/2014-February/026976.html I did add support to pbr for using nested requirements files, and I had posted a PoC for oslo.messaging to allow requirements files for different backends, but some of our CI jobs don't know how to handle that and I never got around to addressing the limitation. - From the packaging perspective, I think you could do a requirements file that basically rolls up requirements.d/*.txt minus test.txt and get all the runtime dependencies that the project knows about, assuming we finished the implementation for this and started using it in projects. I don't really anticipate having time to pursue this in the near future, so if you wanted to pick up the ball and run with it that would be great! :-) Thanks for the back reference! Though I think it's overkill in Neutron case. There, we are not interested in Qpid vs. Rabbitmq issue since we depend on oslo.messaging that is to solve this specific dependency hell. The case of Neutron plugins depending on specific code that is not mentioned in requirements.txt is also a bit different. It's not about alternative implementations of the same library for users to choose from (as it's in case of qpid vs. rabbitmq). Instead, it's plugins here that are really optional, but their dependencies are still strict (=no alternative dependency implementations). So what we need to make optional here are plugins. I've heard that Kyle Mestery was going to propose splitting Neutron codebase in parts, with core plugins sitting in the tree, and vendor specific stuff maintained in separate repos. That would effectively make plugins optional. Till that time, I would make dependencies straight and explicit, putting them in global requirements list. If distributions are interested in not installing optional code for all plugins, they would just separate core from plugins and attach those dependencies to plugin-specific packages (as they already do btw, at least in Red Hat world). As for pip and devstack installations (which are not meant to be the proper way to deploy openstack anyway), I don't see much value in separating requirements. It's ok to have optional libraries installed in your development environment, if only to be able to write unit tests without mocking out the whole underlying library and praying that your assumptions about its API are correct. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT8frcAAoJEC5aWaUY1u57m9oH/0w/RCdJcfZOAwVtGWgd2pH/ BkMpEChlQl40Eb7Vz/9UnLA1jdnnaWmhu9CPEFYs8mYQy/ZMMfB1Ww0e1wl5w0VD pnbepRyYhmmyupZpsG8ywx5lOdSHNWJzUSEsJpK/ZHszgMiZUB/l++mMr+YfjyYK h4SzFpoiz7Dnr3qYfowJQuoeH/yFK3Qd03WAybm39waxj+CX/0DXFVS7xoFFokGa sQWPrMf+xhH2iOVgpgZl2YMzVVDl/bu2NWFkyDepXbTX/gWYI67rx4SbidwWlRYG ifRQbrruV34n6wdBZ8C5s1WonpOpqX6Unay6qxPsn1qmgPKP4M2SckJoZqbvKVk= =A0Gf -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] requirements.txt: explicit vs. implicit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, some plugins depend on modules that are not mentioned in requirements.txt. Among them, Cisco Nexus (ncclient), Brocade (ncclient), Embrane (heleosapi)... Some other plugins put their dependencies in requirements.txt though (like Arista depending on jsonrpclib). There are pros and cons in both cases. The obvious issue with not putting those requirements in the file is that packagers are left uninformed about those implicit requirements existing, meaning plugins are shipped to users with broken dependencies. It also means we ship code that depends on unknown modules grabbed from random places in the internet instead of relying on what's available on pypi, which is a bit scary. With my packager hat on, I would like to suggest to make those dependencies explicit by filling in requirements.txt. This will make packaging a bit easier. Of course, runtime dependencies being set correctly do not mean plugins are working and tested, but at least we give them chance to be tested and used. But, maybe there are valid concerns against doing so. In that case, I would be glad to know how packagers are expected to track those implicit dependencies. I would like to ask community to decide what's the right way to handle those cases. Cheers, /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT7Lu2AAoJEC5aWaUY1u57tDkIAOrx1TWVjke8xxsJtz+tizmg rDgoQyugU8bmaWUFzKi3yVLDFmkOH5iX9RFqj6pgXngydd+cO0Z8CB825uT7kimi tTwTk2o1Ty4lIG38nwi/U8pn+nmzVApjOqtJmBmtZKBtoY7hRUs+QVTz5V5M1AmA MQm0eYZXMQ531k4UTdaFxtZ2xPvnCEsFTWi0vosZLPvccVw33vUnQ0SnewQAgb4w NZ7m302454S2INegqVYlZqQMQXxy6v/BAigyoLXBj8Pl3FsrNU0j3SMtzqSm71ty GCz0qdWckUdgsDFnLyyNXjUV/G9xZ03pYZ5ID2WiVQl5MYbmkAHlJJkjCYIrv3c= =tLTZ -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] requirements.txt: explicit vs. implicit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/14/2014 08:37 AM, Ihar Hrachyshka wrote: Hi all, some plugins depend on modules that are not mentioned in requirements.txt. Among them, Cisco Nexus (ncclient), Brocade (ncclient), Embrane (heleosapi)... Some other plugins put their dependencies in requirements.txt though (like Arista depending on jsonrpclib). There are pros and cons in both cases. The obvious issue with not putting those requirements in the file is that packagers are left uninformed about those implicit requirements existing, meaning plugins are shipped to users with broken dependencies. It also means we ship code that depends on unknown modules grabbed from random places in the internet instead of relying on what's available on pypi, which is a bit scary. With my packager hat on, I would like to suggest to make those dependencies explicit by filling in requirements.txt. This will make packaging a bit easier. Of course, runtime dependencies being set correctly do not mean plugins are working and tested, but at least we give them chance to be tested and used. But, maybe there are valid concerns against doing so. In that case, I would be glad to know how packagers are expected to track those implicit dependencies. I would like to ask community to decide what's the right way to handle those cases. So I raised a similar issue about six months ago and completely failed to follow up on the direction everyone seemed to be onboard with: http://lists.openstack.org/pipermail/openstack-dev/2014-February/026976.html I did add support to pbr for using nested requirements files, and I had posted a PoC for oslo.messaging to allow requirements files for different backends, but some of our CI jobs don't know how to handle that and I never got around to addressing the limitation. - From the packaging perspective, I think you could do a requirements file that basically rolls up requirements.d/*.txt minus test.txt and get all the runtime dependencies that the project knows about, assuming we finished the implementation for this and started using it in projects. I don't really anticipate having time to pursue this in the near future, so if you wanted to pick up the ball and run with it that would be great! :-) - -Ben Cheers, /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT7OS8AAoJEDehGd0Fy7uqtqsH/0kcIB+Q9iA5vR5evC+PDsFb ek+cwvldbgpv0JwFwhgsLtsbRbI9xv1wDpu8L5i30yKzgcPQX5cqYe2WZeG5eCBJ HUHb3t86rCanBU+kp7hpjHSoQbdwhY9gtu1LwiVha/2IeOHchZBzJccxEcACsv0q Es8YkQy3qp9EfegumaL4OHvYFfB/j4NbewxZjAb3mkcpObb6NBM1v+qeubjTEg5I nY8lJMLBXJOLNrR5cg8G7sObh3Cow51GtjwFaiFuZi/9o6whQFXipKnwXkaSRR5U I3YV18sy3NLtoStZdr4/Oa9kUICw1MdDZAckoc5nP+AQeZCBFaPaCPpkLzIcMT8= =CdPl -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev