On Wed, Dec 02, 2015 at 06:44:19AM -0500, Neil Horman wrote: > On Tue, Dec 01, 2015 at 12:37:37PM +0000, Robie Basak wrote: > > Re-sending this unsigned since the ML rejected my signed email. > > > > -1 from Ubuntu without further discussion since it will break us. Please > > don't commit this patch yet. > > > > I don't understand why we must have the complexity of so many shared > > libraries. From a distribution packaging perspective, all I see is that > > this multiplies potential work by twenty times and makes it awkward to > > work with without special tooling (which then needs maintaining). > > > Theres nothing "complex" about the simple fact that a project builds lots of > libraries. Its extreemely common. Any graphic window manager has exactly the > same situation, as do any number of tools that have multiple hardware backends > impelmented in user space (v4l, sane, iptables, to name just a few). > > > Before I go into details, it would be nice if someone could please > > explain why DPDK has to be "special" in needing to do this? I don't > Its not special, see above. Not saying the build environment cant be > improved, > but the fact that there are multiple libraries is pretty straightforward. > > > understand why DPDK must be different to every other userspace library > > out there. If DPDK has a good need to be different then that's fair > > enough. But I feel that if DPDK is deviating from the norm then we need > > to frame the discussion from the perspective of "why DPDK must be > > different", rather than having me trying to explain why the norm is the > > right way to do it. > > > > On Tue, Nov 24, 2015 at 04:31:17PM +0200, Panu Matilainen wrote: > > > That's how Fedora and RHEL are shipping it already and nobody has so much > > > as noticed anything strange, much less complained about it. 20 libraries > > > is but a drop in the ocean on a average distro. > > > > No, it is 20 times the work from the perspective of DPDK package > > maintenance. Let me explain why. > > > > In Debian and Ubuntu, we manage a library transition (an ABI bump in a > > library together with all dependencies moving to use the new ABI) by > > concurrently packaging both the old and new libraries at once. This > > works well with the norm for libraries. We ship one binary package per > > soname, with the major version as part of the package name. This allows > > a system to have two (or more) ABIs installed simultaneously. For a > > library transition, we just package the new version and then that can > > land and work concurrently as we then individually update every > > dependent (library-consuming) package. > So thats, a distribution choice, not an upstream problem. And it seems like a > problem you should have already solved (note the examples above). If you feel > like you need to package multiple ABI versions in the same library, you can, > just update the LIBABIVER of all the libraries, instead of the ones that truly > change, so that each library is guaranteed a newer so version, to make the > library file name unique. Yes you have to make a small change from upstream, > but thats part of the work that distribution maintainers do. > > > > This works because of conventions around sonames, which DPDK breaks > > unless we treat it as twenty different libraries which changes our work > > from easy to painful. > > > > Usually a library transition is managed by hand by the package > > maintainer. It's not taxing because it's straightforward and well > > understood. Update and upload the new ABI source package, then find all > > reverse dependencies and sort them out, recursively. But if the > > maintainer must do it twenty times, then it becomes taxing and prone to > > error. And if the reverse dependency tree differs depending on the split > > library used by library consumers, then it gets far more complex to > > follow. > > > > Admittedly we could tool around this to make it easier, but that's extra > > work (both initially and in maintenance) and prone to error (because > > we'd only be doing it for DPDK). > > > You must already have a solution to this, I can't imagine you package all the > libraries for kde or gnome (or even pam) separately) > > > Packaging a library is usually virtually a no-op in Debian and Ubuntu > > nowadays. Our tooling does it all for us. But packaging DPDK is far from > > this currently because of all this added complexity. From my perspective > > this is unnecessary and makes no sense. We could do all kinds of things > > to work around it (that's what packaging is about) but then we'd have to > > maintain that specialness and I don't see why it must be awkward like > > this instead of just doing it the same way as every other library. > > > > > The combined library as it is simply is no longer a viable option. > > > Besides just being broken (witness the strange hacks people are coming > > > up with to work around issues in it) its ugly because it basically gives > > > the middle finger to all the effort going into version compatibility, > > > and its also big. Few projects will use every library in DPDK, but with > > > the combined library they're forced to lug the 800 pound gorilla along > > > needlessly. > > > > It's broken because it's broken upstream, and that's what we should fix. > > Why is it not viable? How does it give the middle finger to effort going > > into version compatibility? > Because each individual library has a version script that gets applied during > link to version symbols properly. Those scripts dont get applied when > building > the combined library. > > > > Doing it the right way like every other > > userspace library is what *gives us* version compatibility because then > > distributions can straightforwardly install multiple ABI versions at > > once. > Again, Not at all uncommon. You're packaging methodology is the issue here, > not the fact that there are multiple libraries. > > > Finally, I fail to see any "lug the 800 pound gorilla along" saving. We > > (Ubuntu and Fedora) are both shipping all the libraries in one package, > > whether split or combined, so they are all being lugged onto disk > > anyway. Whether split or combined, there is no saving there. And memory > > is hardly saved either because the kernel will just page in and out what > > is needed in both cases. So how does this proposed change give us any > > saving at all? > > > Not true, initalization constructors for PMD's at the very least mean that > every > pmd will get paged in weather you want it or not using the combined library. > Individual libraries let you dynamically load them (via dlopen). I think the > same is true of several other facets of dpdk. > > Neil > I just sent a patch that fixes versioning in combined library, can you please check it: http://dpdk.org/dev/patchwork/patch/9259/
Thanks, ferruh