Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/14 5:15 PM, Ciaran McCreesh wrote: On Thu, 20 Nov 2014 01:36:32 +0100 hasufell hasuf...@gentoo.org wrote: Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. Well the big thing is that unlike Gentoo, Exherbo was able to switch to using Git for its repositories. On top of that, Exherbo also has proper automated tinderbox runs (with automated conflict resolution) for changes, including across repositories, and a much stronger culture of accepting that breaking changes to APIs and APIs that give an error on misuse are necessary for a quality product, and a tolerance of developers making those changes and then applying the fixes to other people's packages. Distributed is much easier to do if you're starting from something which is correct and verified... I'm glad Exherbo has been mentioned - this gives us something specific to discuss, including how it works in practice. Using git is certainly an advantage. Ciaran, could you share more about the automatic tinderbox runs and automated conflict resolution? I look at Exherbo site from time to time but didn't notice this. Please bear with my ignorance, I've even tried searching for things like Exherbo tinderbox. I think you have a good point about necessity of breaking changes from time to time, and APIs that give an error on misuse. This reminds me of these two other good resources: http://www.infoq.com/presentations/effective-api-design (just the slides are at http://www.newt.com/java/GoodApiDesign-JoshBloch.pdf) https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt Note that Linus Torvalds pays very close attention to never break userspace. But within the kernel, large-scale changes are not uncommon, which I think is a good thing. Paweł signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 05:39 AM, hasufell wrote: I see a lot of things to figure out (especially PM-side, tools-side, maybe even PMS, maybe even repository layout, but also documentation and if it makes sense culture-wise), but I don't see a fundamental unsolvable problem here. It would be interesting to see a draft of those proposals. Could you start writing it somewhere on the wiki? Then we can discuss more concrete technical things. Even if it will not result in more distributed model for Gentoo, it could improve our current model of overlays. -- Jauhien signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Thu, 20 Nov 2014 01:36:32 +0100 hasufell hasuf...@gentoo.org wrote: Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. Well the big thing is that unlike Gentoo, Exherbo was able to switch to using Git for its repositories. On top of that, Exherbo also has proper automated tinderbox runs (with automated conflict resolution) for changes, including across repositories, and a much stronger culture of accepting that breaking changes to APIs and APIs that give an error on misuse are necessary for a quality product, and a tolerance of developers making those changes and then applying the fixes to other people's packages. Distributed is much easier to do if you're starting from something which is correct and verified... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 12:41 PM, Jauhien Piatlicki wrote: On 11/20/2014 05:39 AM, hasufell wrote: I see a lot of things to figure out (especially PM-side, tools-side, maybe even PMS, maybe even repository layout, but also documentation and if it makes sense culture-wise), but I don't see a fundamental unsolvable problem here. It would be interesting to see a draft of those proposals. Could you start writing it somewhere on the wiki? Then we can discuss more concrete technical things. Even if it will not result in more distributed model for Gentoo, it could improve our current model of overlays. https://wiki.gentoo.org/wiki/Distributed_Gentoo
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/18/2014 02:12 PM, Jauhien Piatlicki wrote: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. Sounds like there should be some sort of wiki page/guidelines what to keep in mind when creating an overlay. I guess there are two approaches: * make the overlay as independent and consistent as possible * make the overlay as themed and reusable as possible Example: You want to create a games overlay, do you add libsdl, sdl-mixer etc to it? One way to go about it is probably to make a very strong distinction between actual applications and libraries/modules. So you'd rather have two overlays for the above example: games and games-libs? Similar scenarios are: do you include dev-lang/ruby in a ruby overlay, dev-lang/perl in a perl overlay, dev-lang/ghc in a haskell overlay? It's basically mixing toolchain with modules. I think that's why I was previously thinking about submodules. Like breaking an overlay up into smaller parts (without actually creating new overlays), so that people can use specific parts of other overlays without depending on the whole thing. But I'm not sure if that's a good idea and although it's different, it's somewhat similar to just creating a new overlay and depending on it. In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/19/2014 03:36 PM, hasufell wrote: On 11/18/2014 02:12 PM, Jauhien Piatlicki wrote: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. Sounds like there should be some sort of wiki page/guidelines what to keep in mind when creating an overlay. I guess there are two approaches: * make the overlay as independent and consistent as possible * make the overlay as themed and reusable as possible Example: You want to create a games overlay, do you add libsdl, sdl-mixer etc to it? One way to go about it is probably to make a very strong distinction between actual applications and libraries/modules. So you'd rather have two overlays for the above example: games and games-libs? That sounds reasonable. In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. -- Jauhien signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. -- Rich
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, 19 Nov 2014 18:54:05 +0100 hasufell hasuf...@gentoo.org wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. jer
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
Il 20/11/2014 00:58, Rich Freeman ha scritto: On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? somwthing bad The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. I disagree distributed repository does NOT work well, even for release based distro. They are annoing, require magic updating at the right time and break often. Gentoo being compiled can avoid some but not all pitfalls and could introduce a lot more. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). And thinking to be able to impose something more is very optimistic (maybe doable with a much smaller tree like USE=-X @system) If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. -- Rich good luck -- Francesco Riosa
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
Il 20/11/2014 01:00, Jeroen Roovers ha scritto: On Wed, 19 Nov 2014 18:54:05 +0100 hasufell hasuf...@gentoo.org wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. jer worse, mutually incompatibility would be much worse
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 12:58 AM, Rich Freeman wrote: On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? I didn't suggest to regularly fork ebuilds and do things unmodular (see my whole post). I was simply questioning whether all this is actually a big problem and if such a scenario of forked ebuilds that diverge a LOT and cause random build failures in other overlays will happen frequently. The only case this happened to me is when people were eager to hack multilib support into random ebuilds (including games overlays). That was really dangerous. The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. I am not entirely sure if this is just an enhancement or a necessity, but I see your point. But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Thu, 20 Nov 2014 01:36:11 +0100 viv...@gmail.com viv...@gmail.com wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. worse, mutually incompatibility would be much worse I was talking about what is problematic about the act of creating another instance in a different overlay. Not about the well-known problems that ensue afterwards. jer
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, Nov 19, 2014 at 7:36 PM, hasufell hasuf...@gentoo.org wrote: But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). This doesn't really make the issue go away. Are all dependencies going to have to stay in the core? That is a lot of packages, and we'll need lots of controls so that updates to those dependencies don't break all of these distributed repos whose existence we may not even be aware of. Or, maybe we allow the dependencies to be in the overlays. Now we have the issue that there might be 5 different editions of that floating around, and different packages depend on different ones, but they're not intended to co-exist. Oh, and you still have the problem that if the dependency changes it could break anything that uses it. Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros. I'm not sure how Exherbo is doing things, I'm certainly interested in their comments. From what I've read about Nix, it basically installs every package-version into its own directory tree, so if zlib is in 75 different repositories, and has 2 SONAMEs in use, then you'll have up to 150 copies of it installed on your system each in its own directory tree, assuming you have 150 packages each using a different fork of it. I'll optimistically assume that only half of them contain security issues. :) That is my issue with some of these more devops-oriented approaches that try to bypass the centralized distro. They make things really easy for developers, but they seem almost impossible to control from a patching/security/integration standpoint. People have brought up things like Android and iOS and their framework model as a solution, but all the examples of this I've seen are all focused on the presentation layer alone. It is great for a mobile app that doesn't do anything but open sockets back to a server and display widgets. If you're actually building the part of an application that does the real logic/work, then suddenly you find yourself caring about a lot more than list controls. Good luck finding a framework that includes everything in /usr/portage/*-libs -- Rich
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 04:03 AM, Rich Freeman wrote: On Wed, Nov 19, 2014 at 7:36 PM, hasufell hasuf...@gentoo.org wrote: But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). This doesn't really make the issue go away. Are all dependencies going to have to stay in the core? no, why? That is a lot of packages, and we'll need lots of controls so that updates to those dependencies don't break all of these distributed repos whose existence we may not even be aware of. You are exaggerating, imo. We already have these breakages in the tree, because a lot of teams do not communicate, including security bug fixes on libraries that break 20+ packages of a different subsystem (including stable packages, multiple times). Also, no one tests all reverse deps when bumping a package in ~arch. Whether these packages live in one repository or several almost doesn't make any difference, as long as there is communication. It took me 5 minutes to communicate that libsdl2 is not going to be slotted in the tree to all major overlays that had libsdl:2 packaged. They fixed it within 30minutes. That's ~50 times faster than what I expect in the tree. It's not really an issue to have communication channels and a community infrastructure for those things. Sure, these things sound scary if you imagine overlays with 10+ overlay dependencies. But first of all, you don't really depend on an overlay just because you need a few packages from it. If it matters where that package actually comes from, then there is either something wrong with your own package (e.g. because it requires a hacked version of a lib) or something is wrong with the list of high-quality overlays, because some of them ship broken/hacked libraries. It's up to the PM to give suggestions on possible dependencies/overlays that contain the missing package. (that's what paludis already does) Or, maybe we allow the dependencies to be in the overlays. Now we have the issue that there might be 5 different editions of that floating around, and different packages depend on different ones, but they're not intended to co-exist. Oh, and you still have the problem that if the dependency changes it could break anything that uses it. It's debatable whether that is an actual technical problem to be solved or just a reason to mark that overlay low quality. Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros. I'm not sure how Exherbo is doing things, I'm certainly interested in their comments. From what I've read about Nix, it basically installs every package-version into its own directory tree, so if zlib is in 75 different repositories, and has 2 SONAMEs in use, then you'll have up to 150 copies of it installed on your system each in its own directory tree, assuming you have 150 packages each using a different fork of it. I'll optimistically assume that only half of them contain security issues. :) That is my issue with some of these more devops-oriented approaches that try to bypass the centralized distro. They make things really easy for developers, but they seem almost impossible to control from a patching/security/integration standpoint. People have brought up things like Android and iOS and their framework model as a solution, but all the examples of this I've seen are all focused on the presentation layer alone. It is great for a mobile app that doesn't do anything but open sockets back to a server and display widgets. If you're actually building the part of an application that does the real logic/work, then suddenly you find yourself caring about a lot more than list controls. Good luck finding a framework that includes everything in /usr/portage/*-libs You lost me here. It's pretty clear that there will be some more amount of what should my overlay actually be about? involved. I'm sure the maintainer of an overlay will know that 5+ overlays use packages from it and it will very likely become a joint effort, because everyone is interested that the dependencies are ok. I see a lot of things to figure out (especially PM-side, tools-side, maybe even PMS, maybe even repository layout, but also documentation and if it makes sense culture-wise), but I don't see a fundamental unsolvable problem here.
[gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. -- Jauhien signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
Il 18/11/2014 14:12, Jauhien Piatlicki ha scritto: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. As much as I dislike the idea to move development to overlays circular dependancies is not a problem because it's a simple _mutual_ dep. there is not really a concept of before and after at most priority for a package.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/18/2014 03:02 PM, viv...@gmail.com wrote: Il 18/11/2014 14:12, Jauhien Piatlicki ha scritto: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. As much as I dislike the idea to move development to overlays circular dependancies is not a problem because it's a simple _mutual_ dep. there is not really a concept of before and after at most priority for a package. At the moment it is. As `masters` is really not the dependency, but instruction to use eclasses from a given overlay. May be we need to rethink layout.conf a little bit and add real overlay dependencies. But here another question arises: overlays are not specified in PMS and so treated by every PM in a different manner. There master repositories are mentioned, but there is no specification afaik. -- Jauhien signature.asc Description: OpenPGP digital signature
