Re: [gentoo-dev] ROS (Robot Operating System) Overlay for Gentoo
Hi, On Tue, 18 Nov 2014 14:38:12 -0500 Wayne Chang wa...@neverfear.org wrote: Hi All, I got the latest ROS version running on Gentoo and packaged it into an overlay. The closest overlay was ezod (http://www.mavrinac.com/index.cgi?page=gentoo), but it wasn't getting much love. I've been working on this lately; it's not published but I'll try to commit what I have and consider clean enough rather sooner than later :) On the other hand, please CC me on bugs you might open to submit packages. Alexis. PS: you can already remove (and move) your media-libs/collada-dom I think.
Re: [gentoo-dev] Running repoman on the portage tree
On Tue, 18 Nov 2014 21:17:19 -0500 Alec Ten Harmsel a...@alectenharmsel.com wrote: * It took over 4 hours Fun! * So many (~3MB output) warnings, especially upstream parallel compilation bug... thought autoconf handled this, but I guess not autoconf doesn't fix build systems like that. Also, repoman misses a lot of those `(make|MAKEOPTS=).* -j1' strings so your report is incomplete. ;) jer
Re: [gentoo-dev] Running repoman on the portage tree
On 11/19/14 10:17, Alec Ten Harmsel wrote: Hey devs, This is my first mail to this list. If this is out of line, let me know. I've been playing around with Jenkins (continuous integration server) recently for a couple of personal projects, including my own overlay. I thought it would be cool to run 'repoman full' on the portage tree. A couple of things surprised me: * It took over 4 hours That's why I run it in parallel, takes about 45 minutes wall time like that * So many (~3MB output) warnings, especially upstream parallel compilation bug... thought autoconf handled this, but I guess not * 9233 ebuilds that use a deprecated EAPI http://packages.gentooexperimental.org/repoman-checks/ updated per cron job, split by category. Much easier to handle :) Feel free to work on fixing things - there's enough issues that you won't run out of work this decade. Some are easy cosmetic fixes, others will require rewriting the build system from scratch (upstream parallel compilation bug ? postgresql upstream for example doesn't see it as a bug, but patches welcome) http://packages.gentooexperimental.org/repoman-current-issues.txt is a collection of more interesting issues, all of these are also announced to IRC by my messy hack of a shell script by the name of AutoRepoman. #gentoo-bugs, #gentoo-qa on irc.freenode.net Have fun, Patrick
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/18/2014 02:12 PM, Jauhien Piatlicki wrote: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. Sounds like there should be some sort of wiki page/guidelines what to keep in mind when creating an overlay. I guess there are two approaches: * make the overlay as independent and consistent as possible * make the overlay as themed and reusable as possible Example: You want to create a games overlay, do you add libsdl, sdl-mixer etc to it? One way to go about it is probably to make a very strong distinction between actual applications and libraries/modules. So you'd rather have two overlays for the above example: games and games-libs? Similar scenarios are: do you include dev-lang/ruby in a ruby overlay, dev-lang/perl in a perl overlay, dev-lang/ghc in a haskell overlay? It's basically mixing toolchain with modules. I think that's why I was previously thinking about submodules. Like breaking an overlay up into smaller parts (without actually creating new overlays), so that people can use specific parts of other overlays without depending on the whole thing. But I'm not sure if that's a good idea and although it's different, it's somewhat similar to just creating a new overlay and depending on it. In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/19/2014 03:36 PM, hasufell wrote: On 11/18/2014 02:12 PM, Jauhien Piatlicki wrote: On 11/18/2014 04:19 AM, hero...@gentoo.org wrote: Jauhien Piatlicki jauh...@gentoo.org writes: It would be probably good to have in the tree only the core components and move other stuff to the thematic overlays. Then we can have a clear understanding, how things should be: if something is a part of the core system, it goes to the tree, if something is a scientific packages, it lives in the science overlay, if something is a java stuff it lives in the java overlay, etc. This is a good idea. One difficulty: how to handle inter-overlay dependence? Either the dependency should be expressed by overlay + ebuild, or the dependent ebuild should be moved into the core overlay. I haven't figured out a clean solution yet. Yes, this is a weak point of decentralization. We should think how to solve it. The possible solution is using of dependencies between overlays (one overlay says, it depends on other). We already have such a feature, we only need to add more support for it. Example of such a dependency: %cat /var/lib/layman/melpa-stable/metadata/layout.conf repo-name = melpa-stable masters = gnu-elpa gentoo Dependencies on overlays in ebuilds is bad idea I think, as it only will introduce additional problems. Also think what if you need not a package, but an eclass or whatever else. In addition, one question that emerges is possible circular dependencies between overlays. We need a way to handle this. Sounds like there should be some sort of wiki page/guidelines what to keep in mind when creating an overlay. I guess there are two approaches: * make the overlay as independent and consistent as possible * make the overlay as themed and reusable as possible Example: You want to create a games overlay, do you add libsdl, sdl-mixer etc to it? One way to go about it is probably to make a very strong distinction between actual applications and libraries/modules. So you'd rather have two overlays for the above example: games and games-libs? That sounds reasonable. In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. -- Jauhien signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking.
[gentoo-dev] Re: Last rites: app-emacs/csharp-mode
# Ulrich Müller u...@gentoo.org (5 Nov 2014) # Does not build with Emacs 24.4. # Last visible upstream activity in 2011. # Masked for removal in 30 days, bug #528374. app-emacs/csharp-mode Package has been fixed, so it won't be removed. pgp3ETXZh_meX.pgp Description: PGP signature
Re: [gentoo-dev] Portage dependency solving algorithm
Hello, On Mon, 17 Nov 2014 21:55:48 -0800 Zac Medico wrote: On 11/17/2014 09:47 PM, Andrew Savchenko wrote: I use 2.2.14 on both hosts (and usually latest ~x86 portage is there). I thought that running fixpackages should be enough to run emerge with --dynamic-deps=n. It depends on how badly the installed deps have diverged from the corresponding ebuilds in the tree. I tried fixpackages. It fixed some problems and looks like dependencies resolution became faster. But not all problems are fixed and I can't use --dynamic-deps n on both systems for now; and emerge @changed-deps fails due to numerous conflicts, blocks, unsatisfied deps (this is not surprising, since it doesn't try to update all packages in tree). By the way, is there any way to unroll conflict lists in portage output? I mean if I have following: (dev-lang/ghc-7.6.3-r1:0/7.6.3::gentoo, installed) pulled in by =dev-lang/ghc-6.8.2:0/7.6.3= required by (dev-haskell/random-1.0.1.1-r1:0/1.0.1.1::gentoo, installed) ^ (and 68 more with the same problem) How can I see all list of these 68 packages? Sometimes this feature is really desired, e.g. if I don't want to update all @world but need to apply GLSA fix which leads to similar conflicts. I can't find any switch in emerge manual. When I'll manage to run emerge -DNupv @world without errors, I'll send you stats for both runs with and without dynamic deps. Great, hopefully that will reveal some more good things to optimize. By the way, do you need pstats files (e.g. for some extra data) or pdf graphs are sufficient? The pdf graphs are typically enough for me, since they highlight the hot spots really well. I did not even bother with your pstats files. OK. I managed to run emerge -DNupv @world on desktop without conflicts. What was done: 1) fixpacgkages run 2) portage is updated to use your patch from bug 529660 At this point performance boost was really great: from ~35 minutes to ~19-20 minutes. Afterward I tried emerge -DNupv @world with different python versions: (2.7) (~)2.7.8 (3.3) 3.3.5-r1 (3.4) (~)3.4.2 Results are interesting (confidence level for error is 0.95, time real value was used for calculations): 3.3 is 3% ± 5% faster than 2.7 3.4 is 20% ± 5% faster than 3.3 And with python:3.4 and steps above it takes now 15.5 minutes instead of 35. Nice result :) So there is no evidence that portage on 3.3 is faster than on 2.7, but 3.4 is faster than 3.3 with very good confidence. Of course this data is biased by -m cProfile overhead, but bias should similar for each version. Just checked time to run command for python:3.4 without profiling: it takes 11.5 minutes! You may find generated pdf graphs together with system information for each host here: ftp://brunestud.info/gentoo/portage-v2.tar.xz As for hitomi box, it is both slower and have much older packages, so I'm still struggling to fix conflicts and other issues. Results will be available later. P.S. Note for those who would like to use gpro2dot: it should be run with the same python interpreter active as was used during pstats data collection, otherwise it will fail to process data. I spent some time while figuring this out. Best regards, Andrew Savchenko pgpd9Hhzz81ge.pgp Description: PGP signature
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. -- Rich
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, 19 Nov 2014 18:54:05 +0100 hasufell hasuf...@gentoo.org wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. jer
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
Il 20/11/2014 00:58, Rich Freeman ha scritto: On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? somwthing bad The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. I disagree distributed repository does NOT work well, even for release based distro. They are annoing, require magic updating at the right time and break often. Gentoo being compiled can avoid some but not all pitfalls and could introduce a lot more. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). And thinking to be able to impose something more is very optimistic (maybe doable with a much smaller tree like USE=-X @system) If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. -- Rich good luck -- Francesco Riosa
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
Il 20/11/2014 01:00, Jeroen Roovers ha scritto: On Wed, 19 Nov 2014 18:54:05 +0100 hasufell hasuf...@gentoo.org wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. jer worse, mutually incompatibility would be much worse
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 12:58 AM, Rich Freeman wrote: On Wed, Nov 19, 2014 at 12:54 PM, hasufell hasuf...@gentoo.org wrote: On 11/19/2014 06:27 PM, Jauhien Piatlicki wrote: On 11/19/2014 03:36 PM, hasufell wrote: In the end, I'm not sure if this is actually such a big problem. You can still use random ebuilds from random overlays and commit them straight to your own overlay. A bad idea. Bad because of the same reason why copy-past in your code would be bad. Depends. If a third-party overlay dependency regularly breaks my packages, I am going to copy paste it into my own to have more control over it. At that point it is forked. I don't see what's wrong with forking. What happens when 3 overlays all fork the same dependency, and you want to use all three? I didn't suggest to regularly fork ebuilds and do things unmodular (see my whole post). I was simply questioning whether all this is actually a big problem and if such a scenario of forked ebuilds that diverge a LOT and cause random build failures in other overlays will happen frequently. The only case this happened to me is when people were eager to hack multilib support into random ebuilds (including games overlays). That was really dangerous. The distributed repository works well for release-based distros since the core of the OS is fixed. A repository for Ubuntu x.y will always work with Ubuntu x.y, since Ubuntu x.y isn't going to upgrade from libfoo-2 to incompatible libfoo-2.3. On the other hand, libraries on Gentoo can change without warning, and the only quality standard we impose is that the main repo still works (with no forced testing of distributed repos). If we want to truly support first-class distributed repos, then we'll need to impose a number of standards on the main tree that we do not impose today. I am not entirely sure if this is just an enhancement or a necessity, but I see your point. But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros.
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Thu, 20 Nov 2014 01:36:11 +0100 viv...@gmail.com viv...@gmail.com wrote: At that point it is forked. I don't see what's wrong with forking. Forking wouldn't be the problem. Duplication of effort would be the problem. worse, mutually incompatibility would be much worse I was talking about what is problematic about the act of creating another instance in a different overlay. Not about the well-known problems that ensue afterwards. jer
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On 31 October 2014 09:28, Diego Elio Pettenò flamee...@flameeyes.eu wrote: So who wants to pick up the pieces now? Because I'm almost pissed off enough to turn down the tinderbox and give a big FU to Gentoo already. https://bugs.gentoo.org/show_bug.cgi?id=527608 More! https://bugs.gentoo.org/show_bug.cgi?id=529788 Again, is somebody going to stand up and do something or can I shut down my tinderbox and spend my free time playing Baldur's Gate? Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/
Re: [gentoo-dev] Portage dependency solving algorithm
Hi, On Wed, 19 Nov 2014 22:59:05 +0300 Andrew Savchenko wrote: On Mon, 17 Nov 2014 21:55:48 -0800 Zac Medico wrote: [...] When I'll manage to run emerge -DNupv @world without errors, I'll send you stats for both runs with and without dynamic deps. Great, hopefully that will reveal some more good things to optimize. By the way, do you need pstats files (e.g. for some extra data) or pdf graphs are sufficient? The pdf graphs are typically enough for me, since they highlight the hot spots really well. I did not even bother with your pstats files. OK. I managed to run emerge -DNupv @world on desktop without conflicts. What was done: 1) fixpacgkages run 2) portage is updated to use your patch from bug 529660 At this point performance boost was really great: from ~35 minutes to ~19-20 minutes. Afterward I tried emerge -DNupv @world with different python versions: (2.7) (~)2.7.8 (3.3) 3.3.5-r1 (3.4) (~)3.4.2 Results are interesting (confidence level for error is 0.95, time real value was used for calculations): 3.3 is 3% ± 5% faster than 2.7 3.4 is 20% ± 5% faster than 3.3 And with python:3.4 and steps above it takes now 15.5 minutes instead of 35. Nice result :) So there is no evidence that portage on 3.3 is faster than on 2.7, but 3.4 is faster than 3.3 with very good confidence. Of course this data is biased by -m cProfile overhead, but bias should similar for each version. Just checked time to run command for python:3.4 without profiling: it takes 11.5 minutes! You may find generated pdf graphs together with system information for each host here: ftp://brunestud.info/gentoo/portage-v2.tar.xz As for hitomi box, it is both slower and have much older packages, so I'm still struggling to fix conflicts and other issues. Results will be available later. I managed to get data from hitomi too, see: ftp://brunestud.info/gentoo/portage-v3.tar.xz (this archive also contains all graphs previously obtained) Graphs were obtained the same way as on desktop. Portage and python versions are the same, time information follows for _profiled_ runs: (2.7) (~)2.7.8 real55m19.892s user39m11.913s sys 15m37.586s (3.3) 3.3.5-r1 real52m34.640s user36m45.325s sys 15m25.663s (3.4) (~)3.4.2 real53m32.657s user37m12.369s sys 15m52.641s Without profiling using 3.3.5-r1: real25m50.439s user25m28.260s sys 0m7.863s This is quite surprising. On hitomi (Intel Atom N270) there is no difference between 3.3 and 3.4, but both are slightly better than 2.7. (To exclude possible cache issues I made a blank run before first test run.) Probably some arch-dependent optimizations. What surprises me most is that profiling overhead is huge (~105%) compared to overhead on desktop (~35%). CPU speeds are not that different, instruction sets too (Atom has sse2, sse3, ssse3, but lacks 3dnow, 3dnowext). L2 cache is the same (512КB), but L1 differs significantly: 64 KB data/64 KB instruction cache vs 24 KB data/32 KB instruction cache. Look like this is the reason. Best regards, Andrew Savchenko pgp_Znd28i1U3.pgp Description: PGP signature
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On Wed, Nov 19, 2014 at 8:41 PM, Diego Elio Pettenò flamee...@flameeyes.eu wrote: On 31 October 2014 09:28, Diego Elio Pettenò flamee...@flameeyes.eu wrote: So who wants to pick up the pieces now? Because I'm almost pissed off enough to turn down the tinderbox and give a big FU to Gentoo already. https://bugs.gentoo.org/show_bug.cgi?id=527608 More! https://bugs.gentoo.org/show_bug.cgi?id=529788 Again, is somebody going to stand up and do something or can I shut down my tinderbox and spend my free time playing Baldur's Gate? Guys, can we please give the volunteers who are going around uploading logs time to do their jobs before we go closing bugs as invalid? The logs are going to get attached. If somebody is in a mad rush to actually resolve the bug (heaven forbid), then just click on the link. Per the Council meeting: The council recommends that bugs from any developer-run tinderbox not be marked invalid based on whether the logs are attached or pointed to by a permanent URL. The council also encourages efforts to automate the attachment of tinderbox logs to improve the quality of the bugzilla record. Aye: blueness, dberkholz, dilfridge, creffet proxy for radhermit, rich0, ulm, williamh Anybody closing these bugs is going against council recommendation, and we appreciate the efforts of mjo to improve our bug records. Let's not go creating work for ourselves by interfering with efforts to document bugs, and run a tinderbox. I appreciate that the bug wranglers add a GREAT deal of value to our bug reports in general, but honestly I think they would do better to just let the scripts do their jobs here. In the interests of not micromanaging everybody and treating people like children we're trying to avoid having a list of 50,000 policies and rules everybody has to read and follow. With efforts to automatically upload logs, this SHOULD be a moot issue. I'm sure this is all the work of well-intentioned individuals who just didn't hear the news. It probably wouldn't have hurt for me to post the Council summary on the list. Can we put this to rest already? -- Rich
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On Wed, Nov 19, 2014 at 7:36 PM, hasufell hasuf...@gentoo.org wrote: But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). This doesn't really make the issue go away. Are all dependencies going to have to stay in the core? That is a lot of packages, and we'll need lots of controls so that updates to those dependencies don't break all of these distributed repos whose existence we may not even be aware of. Or, maybe we allow the dependencies to be in the overlays. Now we have the issue that there might be 5 different editions of that floating around, and different packages depend on different ones, but they're not intended to co-exist. Oh, and you still have the problem that if the dependency changes it could break anything that uses it. Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros. I'm not sure how Exherbo is doing things, I'm certainly interested in their comments. From what I've read about Nix, it basically installs every package-version into its own directory tree, so if zlib is in 75 different repositories, and has 2 SONAMEs in use, then you'll have up to 150 copies of it installed on your system each in its own directory tree, assuming you have 150 packages each using a different fork of it. I'll optimistically assume that only half of them contain security issues. :) That is my issue with some of these more devops-oriented approaches that try to bypass the centralized distro. They make things really easy for developers, but they seem almost impossible to control from a patching/security/integration standpoint. People have brought up things like Android and iOS and their framework model as a solution, but all the examples of this I've seen are all focused on the presentation layer alone. It is great for a mobile app that doesn't do anything but open sockets back to a server and display widgets. If you're actually building the part of an application that does the real logic/work, then suddenly you find yourself caring about a lot more than list controls. Good luck finding a framework that includes everything in /usr/portage/*-libs -- Rich
Re: [gentoo-dev] [gentoo-project] Re: towards a more distributed model
On 11/20/2014 04:03 AM, Rich Freeman wrote: On Wed, Nov 19, 2014 at 7:36 PM, hasufell hasuf...@gentoo.org wrote: But keep in mind that the core is supposed to shrink with this idea of a distributed model! So it would be less work to actually roll/tag releases than it would be right now (or even do that stuff in branches). This doesn't really make the issue go away. Are all dependencies going to have to stay in the core? no, why? That is a lot of packages, and we'll need lots of controls so that updates to those dependencies don't break all of these distributed repos whose existence we may not even be aware of. You are exaggerating, imo. We already have these breakages in the tree, because a lot of teams do not communicate, including security bug fixes on libraries that break 20+ packages of a different subsystem (including stable packages, multiple times). Also, no one tests all reverse deps when bumping a package in ~arch. Whether these packages live in one repository or several almost doesn't make any difference, as long as there is communication. It took me 5 minutes to communicate that libsdl2 is not going to be slotted in the tree to all major overlays that had libsdl:2 packaged. They fixed it within 30minutes. That's ~50 times faster than what I expect in the tree. It's not really an issue to have communication channels and a community infrastructure for those things. Sure, these things sound scary if you imagine overlays with 10+ overlay dependencies. But first of all, you don't really depend on an overlay just because you need a few packages from it. If it matters where that package actually comes from, then there is either something wrong with your own package (e.g. because it requires a hacked version of a lib) or something is wrong with the list of high-quality overlays, because some of them ship broken/hacked libraries. It's up to the PM to give suggestions on possible dependencies/overlays that contain the missing package. (that's what paludis already does) Or, maybe we allow the dependencies to be in the overlays. Now we have the issue that there might be 5 different editions of that floating around, and different packages depend on different ones, but they're not intended to co-exist. Oh, and you still have the problem that if the dependency changes it could break anything that uses it. It's debatable whether that is an actual technical problem to be solved or just a reason to mark that overlay low quality. Exherbo is already running a more modular approach, I'd be interested what they have to say about this or which problems they were facing. I'll probably also dig a bit deeper into NixOS and see what tools they use for creating NixOS based distros. I'm not sure how Exherbo is doing things, I'm certainly interested in their comments. From what I've read about Nix, it basically installs every package-version into its own directory tree, so if zlib is in 75 different repositories, and has 2 SONAMEs in use, then you'll have up to 150 copies of it installed on your system each in its own directory tree, assuming you have 150 packages each using a different fork of it. I'll optimistically assume that only half of them contain security issues. :) That is my issue with some of these more devops-oriented approaches that try to bypass the centralized distro. They make things really easy for developers, but they seem almost impossible to control from a patching/security/integration standpoint. People have brought up things like Android and iOS and their framework model as a solution, but all the examples of this I've seen are all focused on the presentation layer alone. It is great for a mobile app that doesn't do anything but open sockets back to a server and display widgets. If you're actually building the part of an application that does the real logic/work, then suddenly you find yourself caring about a lot more than list controls. Good luck finding a framework that includes everything in /usr/portage/*-libs You lost me here. It's pretty clear that there will be some more amount of what should my overlay actually be about? involved. I'm sure the maintainer of an overlay will know that 5+ overlays use packages from it and it will very likely become a joint effort, because everyone is interested that the dependencies are ok. I see a lot of things to figure out (especially PM-side, tools-side, maybe even PMS, maybe even repository layout, but also documentation and if it makes sense culture-wise), but I don't see a fundamental unsolvable problem here.
Re: [gentoo-dev] more help needed with gcc-4.8 stabilization, chromium starts heavily using C++11
On Wed, 19 Nov 2014, Diego Elio Pettenò wrote: More! https://bugs.gentoo.org/show_bug.cgi?id=529788 Again, is somebody going to stand up and do something or can I shut down my tinderbox and spend my free time playing Baldur's Gate? http://www.gentoo.org/proj/en/council/meeting-logs/2014-summary.txt The council recommends that bugs from any developer-run tinderbox not be marked invalid based on whether the logs are attached or pointed to by a permanent URL. The council also encourages efforts to automate the attachment of tinderbox logs to improve the quality of the bugzilla record. pgpTMMdD7haf0.pgp Description: PGP signature
[gentoo-portage-dev] [PATCH v3] emerge: check for writable /var/db/pkg (490732)
If there are packages to be merged or unmerge, then bail out early if /var/db/pkg is not writable (in order to avoid a fatal EROFS error which would otherwise occur later on). Behavior remains unchanged for --pretend mode. For --ask mode, it will bail out just after the last relevant --ask prompt. In contrast to the writeable_check module, which operates on files just before they are merged, the new vardbapi.writable check is performed before anything has been built (much earlier). X-Gentoo-Bug: 490732 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=490732 --- PATCH v3 fixes emerge --unmerge --pretend to succeed when /var/db/pkg is not writable (--pretend actions may still be useful when operating with a readonly filesystem). pym/_emerge/actions.py | 15 +++ pym/_emerge/unmerge.py | 9 + pym/portage/dbapi/vartree.py | 12 3 files changed, 36 insertions(+) diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py index 48b0826..6f7dfe0 100644 --- a/pym/_emerge/actions.py +++ b/pym/_emerge/actions.py @@ -428,6 +428,21 @@ def action_build(settings, trees, mtimedb, # least show warnings about missed updates and such. mydepgraph.display_problems() + if not Scheduler._opts_no_self_update.intersection(myopts): + + eroots = set() + for x in mydepgraph.altlist(): + if isinstance(x, Package) and x.operation == merge: + eroots.add(x.root) + + for eroot in eroots: + if not trees[eroot][vartree].dbapi.writable: + writemsg_level(!!! %s\n % + _(Read-only file system: %s) % + trees[eroot][vartree].dbapi._dbroot, + level=logging.ERROR, noiselevel=-1) + return 1 + if (--resume in myopts): favorites=mtimedb[resume][favorites] diff --git a/pym/_emerge/unmerge.py b/pym/_emerge/unmerge.py index df29000..7f6d82a 100644 --- a/pym/_emerge/unmerge.py +++ b/pym/_emerge/unmerge.py @@ -10,6 +10,7 @@ import textwrap import portage from portage import os from portage.dbapi._expand_new_virt import expand_new_virt +from portage.localization import _ from portage.output import bold, colorize, darkgreen, green from portage._sets import SETPREFIX from portage._sets.base import EditablePackageSet @@ -536,6 +537,7 @@ def unmerge(root_config, myopts, unmerge_action, if --pretend in myopts: #we're done... return return os.EX_OK + if --ask in myopts: uq = UserQuery(myopts) if uq.query(Would you like to unmerge these packages?, @@ -546,6 +548,13 @@ def unmerge(root_config, myopts, unmerge_action, print(Quitting.) print() return 128 + signal.SIGINT + + if not vartree.dbapi.writable: + writemsg_level(!!! %s\n % + _(Read-only file system: %s) % vartree.dbapi._dbroot, + level=logging.ERROR, noiselevel=-1) + return 1 + #the real unmerging begins, after a short delay unless we're raging if not unmerge_action == rage-clean and clean_delay and not autoclean: countdown(int(settings[CLEAN_DELAY]), Unmerging) diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py index 0fd1bd9..df031cd 100644 --- a/pym/portage/dbapi/vartree.py +++ b/pym/portage/dbapi/vartree.py @@ -33,6 +33,7 @@ portage.proxy.lazyimport.lazyimport(globals(), 'portage.util.env_update:env_update', 'portage.util.listdir:dircache,listdir', 'portage.util.movefile:movefile', + 'portage.util.path:first_existing', 'portage.util.writeable_check:get_ro_checker', 'portage.util._dyn_libs.PreservedLibsRegistry:PreservedLibsRegistry', 'portage.util._dyn_libs.LinkageMapELF:LinkageMapELF@LinkageMap', @@ -189,6 +190,17 @@ class vardbapi(dbapi): self._cached_counter = None @property + def writable(self): + + Check if var/db/pkg is writable, or permissions are sufficient + to create it if it does not exist yet. + @rtype: bool + @return: True if var/db/pkg is writable or can be created, + False otherwise + + return os.access(first_existing(self._dbroot), os.W_OK) + + @property def root(self): warnings.warn(The root attribute of portage.dbapi.vartree.vardbapi -- 2.0.4
Re: [gentoo-portage-dev] [PATCH v3] emerge: check for writable /var/db/pkg (490732)
On Wed, 19 Nov 2014 07:49:00 -0800 Zac Medico zmed...@gentoo.org wrote: If there are packages to be merged or unmerge, then bail out early if /var/db/pkg is not writable (in order to avoid a fatal EROFS error which would otherwise occur later on). Behavior remains unchanged for --pretend mode. For --ask mode, it will bail out just after the last relevant --ask prompt. In contrast to the writeable_check module, which operates on files just before they are merged, the new vardbapi.writable check is performed before anything has been built (much earlier). X-Gentoo-Bug: 490732 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=490732 --- PATCH v3 fixes emerge --unmerge --pretend to succeed when /var/db/pkg is not writable (--pretend actions may still be useful when operating with a readonly filesystem). LGTM -- Brian Dolbec dolsen
