Re: ports and dependency hell
On Fri, 10 Feb 2017 00:04:27 +0100, Martin Waschbüsch stated: >Thus, what you describe sounds, imho, like wanting to eat the cake >and have it. What good is a goddamn cake if you cannot eat it? What, should I eat someone else's cake instead? Besides, how could you possibly eat a cake if you did not possess it first? Personally, I prefer pie anyway. -- Carmel ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
> Am 09.02.2017 um 18:14 schrieb Julian Elischer : > > Commercial products are Hardly EVER rolling releases. > they lurch from point of stability to point of stability, with large amounts > of testing between releases. >>> On the pkg side of things we need the ability for pkg to say "yeah I >>> know I'm looking for foo-1.2.3.txz as a perfect match, but I've been >>> given some slack on the third digit and I can see a foo-1.2.1.txz, so >>> I'll install that instead". >> I'm not sure how you would do that in a general way that didn't add >> extra work for package maintainers. > pkg would have to do it looking in the pkg index. For every package depending on another as a building block, there are constraints as to which version (or range of versions) can satisfy the dependency. If more than one package depend on the same building blocks but (which is more likely than not) depend on different versions (or ranges of versions) of that package, the constraints become ever stricter, narrowing down the possible combinations. When installing just a limited few packages then maybe knowing about compatible ranges of versions of those packages would allow for more flexibility, but it does not sound practical at all: A specific version of package A can be compiled against all versions of library B v1.1, 1.2 or 1.3. Now, unless the exposed symbols where identical for all three versions, we would need to build a binary package for every combination of A and B. And we have not even considered what other versions of package A people might want to install... REALLY few packages and versions are needed before this becomes totally unfeasible. Now, as I see it, what pkg tries to do is for each snapshot to have as few conflicts as possible when you mix and match arbitrary packages by doing pkg install . I am not saying there are no conflicts, but the system and the package maintainers(!) do a great job of keeping packages both up to date and compatible to each other. You get that stability by not having a lot of choice where versions are concerned. But if you need more control over versions and package options, poudriere offers a very flexible way to create a customized (sub)set of pkgs tailored to your needs. Do the default pkg repo or poudriere cover all possible scenarios? No way! But I fail to understand how they could be expected to do that. Thus, what you describe sounds, imho, like wanting to eat the cake and have it. Martin ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
On 9/2/17 11:02 pm, Steve Wills wrote: Hi Julian, On 02/07/2017 13:03, Julian Elischer wrote: [...] I found this all confusing and vague, but it sounds like what's happening is you need older versions of some software for whatever reason and to provide that you are pulling older versions of ports into a newer ports tree. Is that right? the problem is that the internal APIs of ports are changing too much. Sorry, what do you mean? Can you define "too much" change? the specification of what you are getting is intermixed too much with the makefile components that interact with the .mk files. In a slightly theoretical world you would have two files.. one says "MAJOR=2 ; MINOR=2" (or similar) , and specifies the checksums etc and the other gives options and dependencies (which don't usually change as fast as you think) but uses all the complicted USES stuff for example. so you couth PROBABLY get a range of module versions to compile with the same base makefile as long as it matches the .mk files. If you are going to change the API, then you need to be able to declare the version separately, maybe in a version/distinfo file that can be pulled in separately at a different rev, rather than having it built into the main Makefile of each port. Maybe the Makefile specifies a revision range it can be used with, but that would make a huge improvement right there. The idea of having ports/packages support a version range for dependencies is an interesting one, but I'm not sure how that would work. You can not just slide one port up by 3 months, and another down by 3 months to get the revs one needs because the damned Mk files have changed. If I coudl leave the bulk of the Makefile alone and just slide the 'distingo/rev' file, (or be able to select a rev from one htat gives many alternatives, that woudl be "wonderful". You're better off finding the tree that has the right version of the oldest thing you need to use and then updating the things that you need updated. but that is not reproducible in source control Please, ports framework people, think about how this can be done, If I have to edit a file, the game is lost. Can we please get some understanding from ports people that they are making things REALLY HARD on vendors and it really strengthens the hands of the "ditch FreeBSD and go to Linux" crowd when I need to spend a whole week trying to integrate in a set of 5 new ports into the product. The call "It just works under linux, select the versions you want of each package and type make" is often heard around the company. And management is not totally deaf. If you want to see how its' done better (still not perfect), go build a busybox system. you can effectively select any version of any tool and they all communicate via the pkgconf mechanism and you get the result you want.(mostly). And the API is stable. Sounds like you've got a lot of folks who are used to the "LTS" mindset where they never have to update their software to work with newer versions. But ports is a rolling release in the head branch and quarterly for those branches, and that's how it is going to be unless someone wants to volunteer to maintain branches for longer. The LTS thing works because people are paid to back port fixes and you can get those for free. Commercial products are Hardly EVER rolling releases. they lurch from point of stability to point of stability, with large amounts of testing between releases. On the pkg side of things we need the ability for pkg to say "yeah I know I'm looking for foo-1.2.3.txz as a perfect match, but I've been given some slack on the third digit and I can see a foo-1.2.1.txz, so I'll install that instead". I'm not sure how you would do that in a general way that didn't add extra work for package maintainers. pkg would have to do it looking in the pkg index. Otherwise we just have to spend WAY too much time generating dozens of "matching sets" of packages , that must be kept around forever if just one machine shipped with that set, not to mention the fact that making the matching set is often a real task. Packages should generally be maintained as sets, rather than individual packages, IMHO. See "required by different teams" in the first email. The way to get around the problem above CAN be (not always) to install foo.1.2.1 first and THEN install the package you actually want, and pkg will accept it. The problem comes when pkg needs to install a dependency itself. Then it becomes "super picky", when there is actually a range of package revisions that would do. Instead of letting pkg install what it needs, we need to manually set up scripts to install the dependencies. so that all the work done by pkg is wasted. Please think about these two issues.. You can always use 'pkg lock' to lock a particular version of a package. Steve solution by sledgehammer ___ freebsd-ports@freebsd.org mailing list https://lists.fre
Re: ports and dependency hell
Hi Julian, On 02/07/2017 13:03, Julian Elischer wrote: > This is a serious post on a serious issue that ports framework people > seem unaware of. To be honest, it's kind of a confusing post, at least to me. > It' getting too easy to get into dependency hell here (I've spent the > last week fighting this) > > In a production system we have to choose packages from different eras. > > This is because on an average product different teams are responsible > for different parts of the product, and they all have their own > requirements. Not to mention the stuff that comes in from third party > vendors which "must use version X of bar and Version Z of foo". This is > something that is a fact of life in commercial vendors. Ports needs to > support it, and fast because currently ports is a reason to switch to > Linux. (ammunition for the Linux fanboys). We are only staying for the > ZFS support but that reason will evaporate soon. > > We may need node.js 6.95 and a particular version of an apache mod for > example, and a specific version off npm, which all only appeared in the > ports tree at different times, so either we completely ditch the ports > tree all together and import buildroot2 , which allows every package to > have its own revision (but is Linux centric), or we need to generate > frankenports trees. My curent iteration has 359 different packages, so > you an imagine the hilarity if I need to slide 20 of those back or > forth, all independently. > > There doesn't seem to be any understanding of this fact from the ports > framework, and it means one has to keep one's own ports tree in source > control, as a branch off the FreeBSD one. (maybe I should look at pkgsrc).. I found this all confusing and vague, but it sounds like what's happening is you need older versions of some software for whatever reason and to provide that you are pulling older versions of ports into a newer ports tree. Is that right? > the problem is that the internal APIs of ports are changing too much. Sorry, what do you mean? Can you define "too much" change? > If you are going to change the API, then you need to be able to declare > the version separately, maybe in a version/distinfo file that can be > pulled in separately at a different rev, rather than having it built > into the main Makefile of each port. Maybe the Makefile specifies a > revision range it can be used with, but that would make a huge > improvement right there. The idea of having ports/packages support a version range for dependencies is an interesting one, but I'm not sure how that would work. > You can not just slide one port up by 3 months, and another down by 3 > months to get the revs one needs because the damned Mk files have > changed. If I coudl leave the bulk of the Makefile alone and just slide > the 'distingo/rev' file, (or be able to select a rev from one htat gives > many alternatives, that woudl be "wonderful". You're better off finding the tree that has the right version of the oldest thing you need to use and then updating the things that you need updated. > Please, ports framework people, think about how this can be done, If I > have to edit a file, the game is lost. > > Can we please get some understanding from ports people that they are > making things REALLY HARD on vendors and it really strengthens the hands > of the "ditch FreeBSD and go to Linux" crowd when I need to spend a > whole week trying to integrate in a set of 5 new ports into the product. > > The call "It just works under linux, select the versions you want of > each package and type make" is often heard around the company. And > management is not totally deaf. > > If you want to see how its' done better (still not perfect), go build a > busybox system. you can effectively select any version of any tool and > they all communicate via the pkgconf mechanism and you get the result > you want.(mostly). And the API is stable. Sounds like you've got a lot of folks who are used to the "LTS" mindset where they never have to update their software to work with newer versions. But ports is a rolling release in the head branch and quarterly for those branches, and that's how it is going to be unless someone wants to volunteer to maintain branches for longer. The LTS thing works because people are paid to back port fixes and you can get those for free. > On the pkg side of things we need the ability for pkg to say "yeah I > know I'm looking for foo-1.2.3.txz as a perfect match, but I've been > given some slack on the third digit and I can see a foo-1.2.1.txz, so > I'll install that instead". I'm not sure how you would do that in a general way that didn't add extra work for package maintainers. > Otherwise we just have to spend WAY too much time generating dozens of > "matching sets" of packages , that must be kept around forever if just > one machine shipped with that set, not to mention the fact that making > the matching set is often a real task. Packages should gen
Re: ports and dependency hell
On 08/02/2017 08:03, Julian Elischer wrote: On 8/2/17 3:17 am, Grzegorz Junka wrote: On 07/02/2017 18:03, Julian Elischer wrote: This is a serious post on a serious issue that ports framework people seem unaware of. (...) The call "It just works under linux, select the versions you want of each package and type make" is often heard around the company. And management is not totally deaf. Hi Julian, I may not fully understand how it works but what prevents you from getting sources for the version you want and typing make in them, exactly the way you do it in Linux? It should pick up the versions of dependencies currently installed in the system and compile for them. Is it only when you want to use the ports infrastructure that poses a problem? Nothing stops me from doing that. It's just that means that the ports infrastructure is useless and a complete waste of time right? I'm no ready to admit that, however I may just be in denial. also, downsides to doing that include: * not getting any FreeBSD related fixes (many packages don't work well on FreeBSD without patches) * not being able to play nice with software installed by packages * having a hard time installing FreeBSD specific software that is delivered by ports and pkg as it's primary means of delivery. * having to manually chase package revisions. In areas where I have no expertise. Re 1, I believe you can still apply FreeBSD related patches after you have unpacked sources and before compiling them? Is 2 and 4 from your list somehow easier on Linux than on FreeBSD? As for 3, why would you need to compile them from sources, can't you just install them either from the official or a custom poudriere repository? Grzegorz ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
On Wed, 8 Feb 2017 16:03:56 +0800 Julian Elischer wrote: > On 8/2/17 3:17 am, Grzegorz Junka wrote: > > > > On 07/02/2017 18:03, Julian Elischer wrote: > >> This is a serious post on a serious issue that ports framework > >> people seem unaware of. > >> (...) > >> > >> The call "It just works under linux, select the versions you want > >> of each package and type make" is often heard around the company. > >> And management is not totally deaf. > >> > > > > Hi Julian, > > I may not fully understand how it works but what prevents you from > > getting sources for the version you want and typing make in them, > > exactly the way you do it in Linux? It should pick up the versions > > of dependencies currently installed in the system and compile for > > them. Is it only when you want to use the ports infrastructure that > > poses a problem? > > Nothing stops me from doing that. It's just that means that the ports > infrastructure is useless and a complete waste of time right? > I'm no ready to admit that, however I may just be in denial. It wasn't entirely clear what you were comparing FreeBSD ports with, is it specifically buildroot2? ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
On 02/08/17 03:03, Julian Elischer wrote: > [...] > I'm discouraged to not hear back from any of the ports 'committee'. > [...] Possibly because this has been the topic of a number of rancorous mail threads in the last few months already, and everybody is fatigued. What there *hasn't* been is a consensus on what to do about it. And with all respect perhaps that's inevitable in a vibrant and active open-source project run entirely by volunteers. I wish I could omnisciently propound the right thing to do(tm) so persuasively that everybody would immediately set to work on it. But I can't, even in my own overactive imagination. -- George signature.asc Description: OpenPGP digital signature
Re: ports and dependency hell
Hi! > I'm discouraged to not hear back from any of the ports 'committee'. I'm not from the committee 8-), but I think you raise relevant points. It is not easy to cover them. All of the pkg-developers have their tables full of work, so ... Crafting a well-reflected answer takes time. -- p...@opsec.eu+49 171 3101372 3 years to go ! ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
On 8/2/17 3:17 am, Grzegorz Junka wrote: On 07/02/2017 18:03, Julian Elischer wrote: This is a serious post on a serious issue that ports framework people seem unaware of. (...) The call "It just works under linux, select the versions you want of each package and type make" is often heard around the company. And management is not totally deaf. Hi Julian, I may not fully understand how it works but what prevents you from getting sources for the version you want and typing make in them, exactly the way you do it in Linux? It should pick up the versions of dependencies currently installed in the system and compile for them. Is it only when you want to use the ports infrastructure that poses a problem? Nothing stops me from doing that. It's just that means that the ports infrastructure is useless and a complete waste of time right? I'm no ready to admit that, however I may just be in denial. also, downsides to doing that include: * not getting any FreeBSD related fixes (many packages don't work well on FreeBSD without patches) * not being able to play nice with software installed by packages * having a hard time installing FreeBSD specific software that is delivered by ports and pkg as it's primary means of delivery. * having to manually chase package revisions. In areas where I have no expertise. upsides include the ability to use autotools etc as they are designed to be used and pkgconf to knit everything together without worring about version problems. (downside is using autotools :-) ) So there is a price to pay each way.. I'm discouraged to not hear back from any of the ports 'committee'. Grzegorz ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org" ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: ports and dependency hell
On 07/02/2017 18:03, Julian Elischer wrote: This is a serious post on a serious issue that ports framework people seem unaware of. (...) The call "It just works under linux, select the versions you want of each package and type make" is often heard around the company. And management is not totally deaf. Hi Julian, I may not fully understand how it works but what prevents you from getting sources for the version you want and typing make in them, exactly the way you do it in Linux? It should pick up the versions of dependencies currently installed in the system and compile for them. Is it only when you want to use the ports infrastructure that poses a problem? Grzegorz ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
ports and dependency hell
This is a serious post on a serious issue that ports framework people seem unaware of. It' getting too easy to get into dependency hell here (I've spent the last week fighting this) In a production system we have to choose packages from different eras. This is because on an average product different teams are responsible for different parts of the product, and they all have their own requirements. Not to mention the stuff that comes in from third party vendors which "must use version X of bar and Version Z of foo". This is something that is a fact of life in commercial vendors. Ports needs to support it, and fast because currently ports is a reason to switch to Linux. (ammunition for the Linux fanboys). We are only staying for the ZFS support but that reason will evaporate soon. We may need node.js 6.95 and a particular version of an apache mod for example, and a specific version off npm, which all only appeared in the ports tree at different times, so either we completely ditch the ports tree all together and import buildroot2 , which allows every package to have its own revision (but is Linux centric), or we need to generate frankenports trees. My curent iteration has 359 different packages, so you an imagine the hilarity if I need to slide 20 of those back or forth, all independently. There doesn't seem to be any understanding of this fact from the ports framework, and it means one has to keep one's own ports tree in source control, as a branch off the FreeBSD one. (maybe I should look at pkgsrc).. the problem is that the internal APIs of ports are changing too much. If you are going to change the API, then you need to be able to declare the version separately, maybe in a version/distinfo file that can be pulled in separately at a different rev, rather than having it built into the main Makefile of each port. Maybe the Makefile specifies a revision range it can be used with, but that would make a huge improvement right there. You can not just slide one port up by 3 months, and another down by 3 months to get the revs one needs because the damned Mk files have changed. If I coudl leave the bulk of the Makefile alone and just slide the 'distingo/rev' file, (or be able to select a rev from one htat gives many alternatives, that woudl be "wonderful". Please, ports framework people, think about how this can be done, If I have to edit a file, the game is lost. Can we please get some understanding from ports people that they are making things REALLY HARD on vendors and it really strengthens the hands of the "ditch FreeBSD and go to Linux" crowd when I need to spend a whole week trying to integrate in a set of 5 new ports into the product. The call "It just works under linux, select the versions you want of each package and type make" is often heard around the company. And management is not totally deaf. If you want to see how its' done better (still not perfect), go build a busybox system. you can effectively select any version of any tool and they all communicate via the pkgconf mechanism and you get the result you want.(mostly). And the API is stable. On the pkg side of things we need the ability for pkg to say "yeah I know I'm looking for foo-1.2.3.txz as a perfect match, but I've been given some slack on the third digit and I can see a foo-1.2.1.txz, so I'll install that instead". Otherwise we just have to spend WAY too much time generating dozens of "matching sets" of packages , that must be kept around forever if just one machine shipped with that set, not to mention the fact that making the matching set is often a real task. The way to get around the problem above CAN be (not always) to install foo.1.2.1 first and THEN install the package you actually want, and pkg will accept it. The problem comes when pkg needs to install a dependency itself. Then it becomes "super picky", when there is actually a range of package revisions that would do. Instead of letting pkg install what it needs, we need to manually set up scripts to install the dependencies. so that all the work done by pkg is wasted. Please think about these two issues.. Julian ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
