Re: BEWARE: a problematic glibc made it to stable (F16)
Richard W.M. Jones wrote: I'm not convinced yet this is a glibc issue. It could be a problem in the threaded work-queue code in git-grep which is just exposed by the change in glibc. No one will know until we finally diagnose the bug. The analysis in the bug is now that this is indeed a bug in the glibc headers, which can poison builds of pretty much ANY threaded package. This sounds very bad, it means an unknown number of packages may have to be rebuilt to really fix the problem. But even without that, breaking the SCM Fedora itself uses (along with a growing number of upstream projects) in a Fedora release sounds very much unacceptable. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, Oct 24, 2011 at 10:46:31AM -0700, Adam Williamson wrote: On Mon, 2011-10-24 at 18:50 +0200, Kevin Kofler wrote: Adam Williamson wrote: We have lots of suggestions. As I've said at least fifty times, it's pointless going too far with the slapping of band-aids on the current karma system, because it's fundamentally too simplistic: it's never going to be perfect and there is a definite point of diminishing returns if we keep screwing with it. Right. That's why we need to abolish it. Why? How would that make anything better? With the proven tester system, one somewhat-broken update got through. Without it, we would have had five or six utterly broken glibc updates this F16 cycle. Just check the history of submissions to glibc in Bodhi. Given the known attitudes of the glibc maintainers, if they were allowed to simply submit all their builds directly to stable, they would certainly have done so...and broken everyone's systems time and time again. I'd say the history of F16 updates to glibc demonstrates the raging success of the proventesters system, not its failure. You snipped the part where Kevin wrote [...] if the maintainer demonstrates incompetence at taking these decisions, the offending maintainer needs to be replaced. The problem here appears to be a human one, not something that software is going to fix any time soon. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, Oct 24, 2011 at 11:34:46PM +0200, Jim Meyering wrote: Jim Meyering wrote: Adam Williamson wrote: ... The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. The problem is bigger than it first seemed, and still not fixed. True, I noticed the problem initially when running a just-built git, but in fact the distributed /usr/bin/git demonstrates precisely the same heap corruption as the one I built. See the further discussion on http://bugzilla.redhat.com/747377 The underlying bug seems pthread-related. When I make git grep run without using threads there is no problem. To demonstrate the problem, run this on a multi-core system, in a clone of a decent-sized git repository like git's own: for i in $(seq 100);do echo $i; timeout 1 ./git grep -q stat;done Oops. Drop the ./, of course, to test /usr/bin/grep: I think you mean /usr/libexec/git-core/git-grep :-) I'm not convinced yet this is a glibc issue. It could be a problem in the threaded work-queue code in git-grep which is just exposed by the change in glibc. No one will know until we finally diagnose the bug. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Tue, 2011-10-25 at 08:32 +0100, Richard W.M. Jones wrote: You snipped the part where Kevin wrote [...] if the maintainer demonstrates incompetence at taking these decisions, the offending maintainer needs to be replaced. The problem here appears to be a human one, not something that software is going to fix any time soon. Sure. But in the meantime, the software sure is mitigating the problem. (Getting humans replaced is something of an arduous process, in Fedora.) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On 10/23/2011 07:47 PM, Henrik Nordström wrote: Disable automatic push to stable when there is any negative karma, requiring the package maintainer to initiate the push even if karma kriteria have been met. This idea has been suggested: https://fedorahosted.org/bodhi/ticket/618 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Sun, 2011-10-23 at 04:14 +0200, Kevin Kofler wrote: The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It just hinders getting legitimate fixes out and does nothing to stop regressions. glibc is even critpath, yet broken crap still goes out. Except...12.999 got pushed out precisely *because* it fixed the most critical breakage in 12. I'm sure you've previously argued that 'legitimate fixes' should go out even if they break something else, so why are you complaining when it happens? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, 2011-10-24 at 02:47 +0200, Henrik Nordström wrote: Don't automatically push to stable until at least X days (3?) have passed, enabling sufficient time for regressions to be detected. Package maintainer can initiate push earlier by Push to stable if needed and confident there is no regressions. This would cause significant problems around crunch times. We would wind up having to have releng super-push far more updates because we simply don't always *have* three days to wait to hit deadlines. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Sun, 2011-10-23 at 17:04 -0500, Rex Dieter wrote: The fail(*), imo, was with 12.999 going stable containing known-regressions. So, any suggestions, if any, to prevent any similar series of events? We have lots of suggestions. As I've said at least fifty times, it's pointless going too far with the slapping of band-aids on the current karma system, because it's fundamentally too simplistic: it's never going to be perfect and there is a definite point of diminishing returns if we keep screwing with it. What we need is the non-numeric karma system which Bodhi 2.0 is supposed to be bringing in. No amount of tweaking with the rules of Bodhi 1.0 is going to Magically Solve Everything, because '1, 0, -1' is simply too limited a vocabulary to express everything we need to express about updates. While we're fiddling, though, I do think negative karma should have more value than it currently does. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Sun, 2011-10-23 at 23:43 -0700, Adam Williamson wrote: On Sun, 2011-10-23 at 04:14 +0200, Kevin Kofler wrote: The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It just hinders getting legitimate fixes out and does nothing to stop regressions. glibc is even critpath, yet broken crap still goes out. Except...12.999 got pushed out precisely *because* it fixed the most critical breakage in 12. I'm sure you've previously argued that 'legitimate fixes' should go out even if they break something else, so why are you complaining when it happens? Oh - and remember, the goal of the critpath process is to ensure we don't send out updates that break people's systems. It worked fine in this case: no glibc update which breaks systems was approved. All the ones which caused major runtime breakage got rejected. The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Sun, Oct 23, 2011 at 4:14 AM, Kevin Kofler kevin.kof...@chello.at wrote: Jim Meyering wrote: glibc-2.14.90-12.999, which has just made it to stable provokes a hard-to-diagnose (for me at least) problem. While most things work, and it fixed two problems that affected me, it caused me some frustration: https//bugzilla.redhat.com/747377 glibc-2.14.90-12.999 also breaks the build of ANY C++ code using fenv.h, which affects at least Qt (but likely also several other C++ packages, particularly mathematical ones, but not only, as can be seen from Qt). Thankfully, that showstopper is fixed in -13 which is already in stable by now (because it was aggressively up-karma'd by the KDE SIG). The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It just hinders getting legitimate fixes out and does nothing to stop regressions. glibc is even critpath, yet broken crap still goes out. Yes and doing less testing will prevent the crap from getting out ... oh wait See this does not make much sense so stop repeating that. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Sun, Oct 23, 2011 at 05:04:48PM -0500, Rex Dieter wrote: So, any suggestions, if any, to prevent any similar series of events? Do the development in Rawhide and cherry pick only well-tested bug fix commits to the stable branch (F16 in this case). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On 10/24/2011 02:47 AM, Henrik Nordström wrote: sön 2011-10-23 klockan 17:04 -0500 skrev Rex Dieter: The fail(*), imo, was with 12.999 going stable containing known-regressions. So, any suggestions, if any, to prevent any similar series of events? My suggestions: Disable automatic push to stable when there is any negative karma, requiring the package maintainer to initiate the push even if karma kriteria have been met. Don't automatically push to stable until at least X days (3?) have passed, enabling sufficient time for regressions to be detected. Package maintainer can initiate push earlier by Push to stable if needed and confident there is no regressions. Make push to stable require extra confirmation when there is negative karma, making sure that whoever is initiating the push have actually looked at why there is negative karma. Regards Henrik I don't think that any policy, which was introduced to prevent broken glibc updates helped. glibc can broke many packages, so it's perfect candidate for automatic tests, which should include even rebuilds. I wouldn't punish other people who test their packages properly or even wrote a test suite. Marcela -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
sön 2011-10-23 klockan 23:45 -0700 skrev Adam Williamson: This would cause significant problems around crunch times. We would wind up having to have releng super-push far more updates because we simply don't always *have* three days to wait to hit deadlines. note that I only proposed automatic push to be delayed somewhat, not manually initiated push by the package / update request owner. Regards Henrik -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Once upon a time, Adam Williamson awill...@redhat.com said: Oh - and remember, the goal of the critpath process is to ensure we don't send out updates that break people's systems. It worked fine in this case: no glibc update which breaks systems was approved. All the ones which caused major runtime breakage got rejected. The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. That's splitting hairs; you're assuming that no Fedora users use their system to compile things. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, 2011-10-24 at 11:06 +0200, Henrik Nordström wrote: sön 2011-10-23 klockan 23:45 -0700 skrev Adam Williamson: This would cause significant problems around crunch times. We would wind up having to have releng super-push far more updates because we simply don't always *have* three days to wait to hit deadlines. note that I only proposed automatic push to be delayed somewhat, not manually initiated push by the package / update request owner. We rely on autopush in many cases; package maintainers go home sometimes. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, 2011-10-24 at 09:51 -0500, Chris Adams wrote: Once upon a time, Adam Williamson awill...@redhat.com said: Oh - and remember, the goal of the critpath process is to ensure we don't send out updates that break people's systems. It worked fine in this case: no glibc update which breaks systems was approved. All the ones which caused major runtime breakage got rejected. The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. That's splitting hairs; you're assuming that no Fedora users use their system to compile things. No, I'm not. Being unable to compile things is an inconvenience that's easily remedied by five minutes looking at a mailing list. Being unable to boot to a working desktop is a massive stinking problem. They're both problems, sure. But one is a much bigger problem than the other. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Adam Williamson wrote: We have lots of suggestions. As I've said at least fifty times, it's pointless going too far with the slapping of band-aids on the current karma system, because it's fundamentally too simplistic: it's never going to be perfect and there is a definite point of diminishing returns if we keep screwing with it. Right. That's why we need to abolish it. What we need is the non-numeric karma system which Bodhi 2.0 is supposed to be bringing in. No amount of tweaking with the rules of Bodhi 1.0 is going to Magically Solve Everything, because '1, 0, -1' is simply too limited a vocabulary to express everything we need to express about updates. Making the system more complex will only make it more broken, not less. You just cannot predict all the possible kinds of feedback which come in. At the current state of the art of technology, only a human can make this decision correctly, so we should let a human, the maintainer, take it. And if the maintainer demonstrates incompetence at taking these decisions, the offending maintainer needs to be replaced. No amount of software can fix incompetence. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Mon, 2011-10-24 at 18:50 +0200, Kevin Kofler wrote: Adam Williamson wrote: We have lots of suggestions. As I've said at least fifty times, it's pointless going too far with the slapping of band-aids on the current karma system, because it's fundamentally too simplistic: it's never going to be perfect and there is a definite point of diminishing returns if we keep screwing with it. Right. That's why we need to abolish it. Why? How would that make anything better? With the proven tester system, one somewhat-broken update got through. Without it, we would have had five or six utterly broken glibc updates this F16 cycle. Just check the history of submissions to glibc in Bodhi. Given the known attitudes of the glibc maintainers, if they were allowed to simply submit all their builds directly to stable, they would certainly have done so...and broken everyone's systems time and time again. I'd say the history of F16 updates to glibc demonstrates the raging success of the proventesters system, not its failure. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Adam Williamson wrote: ... The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. The problem is bigger than it first seemed, and still not fixed. True, I noticed the problem initially when running a just-built git, but in fact the distributed /usr/bin/git demonstrates precisely the same heap corruption as the one I built. See the further discussion on http://bugzilla.redhat.com/747377 The underlying bug seems pthread-related. When I make git grep run without using threads there is no problem. To demonstrate the problem, run this on a multi-core system, in a clone of a decent-sized git repository like git's own: for i in $(seq 100);do echo $i; timeout 1 ./git grep -q stat;done -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Jim Meyering wrote: Adam Williamson wrote: ... The only breakage in one which was approved was to do with compiling things - which, sure, is a pain in the ass, but it's not the kind of problem critpath was introduced to deal with in the first place. The problem is bigger than it first seemed, and still not fixed. True, I noticed the problem initially when running a just-built git, but in fact the distributed /usr/bin/git demonstrates precisely the same heap corruption as the one I built. See the further discussion on http://bugzilla.redhat.com/747377 The underlying bug seems pthread-related. When I make git grep run without using threads there is no problem. To demonstrate the problem, run this on a multi-core system, in a clone of a decent-sized git repository like git's own: for i in $(seq 100);do echo $i; timeout 1 ./git grep -q stat;done Oops. Drop the ./, of course, to test /usr/bin/grep: for i in $(seq 100);do echo $i; timeout 1 git grep -q stat;done -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Am 23.10.2011 04:14, schrieb Kevin Kofler: Jim Meyering wrote: glibc-2.14.90-12.999, which has just made it to stable provokes a hard-to-diagnose (for me at least) problem. While most things work, and it fixed two problems that affected me, it caused me some frustration: https//bugzilla.redhat.com/747377 glibc-2.14.90-12.999 also breaks the build of ANY C++ code using fenv.h, which affects at least Qt (but likely also several other C++ packages, particularly mathematical ones, but not only, as can be seen from Qt). Thankfully, that showstopper is fixed in -13 which is already in stable by now (because it was aggressively up-karma'd by the KDE SIG). The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It just hinders getting legitimate fixes out and does nothing to stop regressions. glibc is even critpath, yet broken crap still goes out. Since a few days I've got the problem that lazarus and applications written with lazarus are running into an exception on closing them. Originaly I thought that this is a problem of that latest gtk2 update but could it be that glibc also affects lazarus, even if is pascal? Regards Heiko -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Kevin Kofler wrote: Jim Meyering wrote: glibc-2.14.90-12.999, which has just made it to stable provokes a hard-to-diagnose (for me at least) problem. While most things work, and it fixed two problems that affected me, it caused me some frustration: https//bugzilla.redhat.com/747377 glibc-2.14.90-12.999 also breaks the build of ANY C++ code using fenv.h, which affects at least Qt (but likely also several other C++ packages, particularly mathematical ones, but not only, as can be seen from Qt). Thankfully, that showstopper is fixed in -13 which is already in stable by now (because it was aggressively up-karma'd by the KDE SIG). The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It worked in a way, but was an interesting case. As I see it, the general series of events went something like this: -10 was *generally* ok -11 fixed a few bugs, but introduced the nasty breaks grokking of /etc/groups, down karma'd -12 fixed some other bug, down-karma'd for not fixing -11 -12.999 got massive up-karma for fixing -11 problem, *but* introduced a new problem/regression. Got pushed stable. -13 was, of course, damage control from the fall-out from 12.999... Now, I'd argue the process worked up to -12.999, regressions were kept out of stable updates. The fail(*), imo, was with 12.999 going stable containing known-regressions. So, any suggestions, if any, to prevent any similar series of events? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
sön 2011-10-23 klockan 17:04 -0500 skrev Rex Dieter: The fail(*), imo, was with 12.999 going stable containing known-regressions. So, any suggestions, if any, to prevent any similar series of events? My suggestions: Disable automatic push to stable when there is any negative karma, requiring the package maintainer to initiate the push even if karma kriteria have been met. Don't automatically push to stable until at least X days (3?) have passed, enabling sufficient time for regressions to be detected. Package maintainer can initiate push earlier by Push to stable if needed and confident there is no regressions. Make push to stable require extra confirmation when there is negative karma, making sure that whoever is initiating the push have actually looked at why there is negative karma. Regards Henrik -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Jim Meyering wrote: glibc-2.14.90-12.999, which has just made it to stable provokes a hard-to-diagnose (for me at least) problem. While most things work, and it fixed two problems that affected me, it caused me some frustration: https//bugzilla.redhat.com/747377 glibc-2.14.90-12.999 also breaks the build of ANY C++ code using fenv.h, which affects at least Qt (but likely also several other C++ packages, particularly mathematical ones, but not only, as can be seen from Qt). Thankfully, that showstopper is fixed in -13 which is already in stable by now (because it was aggressively up-karma'd by the KDE SIG). The fact that a glibc with showstoppers of this kind got pushed to stable shows that the karma system does not work at all. It just hinders getting legitimate fixes out and does nothing to stop regressions. glibc is even critpath, yet broken crap still goes out. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Fri, Oct 21, 2011 at 1:21 AM, Rahul Sundaram methe...@gmail.com wrote: I'd vote for #1, but that's a much longer conversation that should be had upstream and before we even get close to bringing it to FESCo. FESCo is the entity which can have that conversation with Glibc upstream on behalf of Fedora. Who else can? The Glibc package maintainer. I'm pretty sure he understands upstream, and FESCo should probably start the discussion with him first anyway. josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Josh Boyer jwbo...@gmail.com writes: On Fri, Oct 21, 2011 at 1:21 AM, Rahul Sundaram methe...@gmail.com wrote: FESCo is the entity which can have that conversation with Glibc upstream on behalf of Fedora. Who else can? The Glibc package maintainer. I'm pretty sure he understands upstream, and FESCo should probably start the discussion with him first anyway. I'm not exactly sure what glibc upstream (defined as people without commit rights to Fedora git) have to do with this at all. The issue AIUI is that unproven glibc changes are getting committed to stable Fedora branches rather than rawhide where they belong. Surely, this is a matter to discuss with the Fedora maintainer(s) of glibc and nobody else. And yes, I think it's about time for FESCo to step in and lay down the law. regards, tom lane -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
The Glibc package maintainer. I'm pretty sure he understands upstream, and FESCo should probably start the discussion with him first anyway. I'm not exactly sure what glibc upstream (defined as people without commit rights to Fedora git) have to do with this at all. The issue AIUI is that unproven glibc changes are getting committed to stable Fedora branches rather than rawhide where they belong. Surely, this is a matter to discuss with the Fedora maintainer(s) of glibc and nobody else. They are mostly the same people. Its not like glibc is that inviting for a development community to gather around. Dave. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Fri, Oct 21, 2011 at 1:55 PM, David Airlie airl...@redhat.com wrote: The Glibc package maintainer. I'm pretty sure he understands upstream, and FESCo should probably start the discussion with him first anyway. I've started a dialog with the glibc packager and explained the concerns I'm seeing. -- Jared Smith Fedora Project Leader -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 2011-10-19 at 20:35 -0400, Josh Boyer wrote: Except that Fedora _has_ been glibc's development platform for as long as I can remember. The Fedora project might not think so, but it's exactly what upstream glibc does. Indeed, this has been the case since it was still called Red Hat Linux. - ajax signature.asc Description: This is a digitally signed message part -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On 10/20/2011 06:05 AM, Josh Boyer wrote: Except that Fedora _has_ been glibc's development platform for as long as I can remember. The Fedora project might not think so, but it's exactly what upstream glibc does. I am aware of this but our policies have changed and either they need to get a exception and we should stop this. We can't let individual maintainers, regardless of legacy, override our consensus on how Fedora should work without any consideration of how the distribution is put together. It is disrespectful of contributors who do follow the policy even if they have disagreements with it. I'd vote for #1, but that's a much longer conversation that should be had upstream and before we even get close to bringing it to FESCo. FESCo is the entity which can have that conversation with Glibc upstream on behalf of Fedora. Who else can? Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 2011-10-19 at 20:51 +0200, Jim Meyering wrote: I spent too many hours debugging this today, so feel obliged to warn about this. Plus, I feel a little guilty for giving it positive karma initially. Today's -1 was too late. glibc-2.14.90-12.999, which has just made it to stable provokes a hard-to-diagnose (for me at least) problem. While most things work, and it fixed two problems that affected me, it caused me some frustration: https//bugzilla.redhat.com/747377 TL;DR: while most things worked fine, and gcc even bootstrapped and passed most of make check (I did that when I was wondering if I had bad RAM), this version of glibc appears to make it so when you compile git from cloned sources, the resulting git program evokes double frees, arbitrary heap corruption, aborts, hangs, weird incomplete read errors, etc. But only when you compile with -O2, not with -O1 or less. Now, you might think that this is all git's fault, and maybe glibc is merely exposing it. That may well be true. Until we find the underlying cause we won't know for sure. However, I was surprised to see that valgrind reported nothing, time after time, while glibc was obviously detecting heap corruption. To recover an F16 system that works better, I ran this: yum downgrade glibc glibc-static glibc-devel glibc-common glibc-headers \ glibc-utils nscd What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( Simo. -- Simo Sorce * Red Hat, Inc * New York -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice, but then, I'd also like a solid gold toilet and that doesn't appear to be on the verge of showing up, either... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Simo Sorce wrote: On Wed, 2011-10-19 at 20:51 +0200, Jim Meyering wrote: ... To recover an F16 system that works better, I ran this: yum downgrade glibc glibc-static glibc-devel glibc-common glibc-headers \ glibc-utils nscd What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( That command brought me back to glibc-2.14.90-10.x86_64 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, Oct 19, 2011 at 12:36:36 -0700, Adam Williamson awill...@redhat.com wrote: You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice, but then, I'd also like a solid gold toilet and that doesn't appear to be on the verge of showing up, either... Wouldn't that be kind of cold? I would think a material that is a poor heat conductor would be a better choice. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 2011-10-19 at 15:25 -0500, Bruno Wolff III wrote: On Wed, Oct 19, 2011 at 12:36:36 -0700, Adam Williamson awill...@redhat.com wrote: You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice, but then, I'd also like a solid gold toilet and that doesn't appear to be on the verge of showing up, either... Wouldn't that be kind of cold? I would think a material that is a poor heat conductor would be a better choice. Hehe. It's from some old comic strip, I think, and it's just always stuck in my mind as my go-to Random Idle Wish thing. I think it might have been an old Charlie Brooker comic actually. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
Am 19.10.2011 23:09, schrieb Richard W.M. Jones: On Wed, Oct 19, 2011 at 12:36:36PM -0700, Adam Williamson wrote: On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice, but then, I'd also like a solid gold toilet and that doesn't appear to be on the verge of showing up, either... +1000 Why are we putting glibc git snapshots into Fedora 16, just days before the final release? IMHO Rawhide should be the only place where version-control-snapshots of such an important component like glibc should be allowed. Maybe it would be better to let the value of positive karma depend on the severity of the package. That would mean that packages like glibc would require more positive karma for being pushed to stable than packages like gedit. Regards, Heiko -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 2011-10-19 at 23:36 +0200, Heiko Adams wrote: Am 19.10.2011 23:09, schrieb Richard W.M. Jones: On Wed, Oct 19, 2011 at 12:36:36PM -0700, Adam Williamson wrote: On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice, but then, I'd also like a solid gold toilet and that doesn't appear to be on the verge of showing up, either... +1000 Why are we putting glibc git snapshots into Fedora 16, just days before the final release? IMHO Rawhide should be the only place where version-control-snapshots of such an important component like glibc should be allowed. Maybe it would be better to let the value of positive karma depend on the severity of the package. That would mean that packages like glibc would require more positive karma for being pushed to stable than packages like gedit. We haven't really had any karma problems with glibc. All the really bad glibcs have been correctly rejected. 12.999 was accepted, but at least its issues were only in compilation rather than at runtime. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On 10/20/2011 01:06 AM, Adam Williamson wrote: On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice It is horrendously stupid to do this. It is not a question of being kind or nice. Fedora is not glibc's development platform. They should get their own and stop mucking around and breaking things unnecessarily and repeatedly. If they won't stop on their own, FESCo must push them to stop. Rahul -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, 19 Oct 2011 23:36:43 +0200, HA (Heiko) wrote: IMHO Rawhide should be the only place where version-control-snapshots of such an important component like glibc should be allowed. Maybe it would be better to let the value of positive karma depend on the severity of the package. That would mean that packages like glibc would require more positive karma for being pushed to stable than packages like gedit. Bodhi would need to be changed first. So far, what some package maintainers do is to wait for a first +1, then edit the bodhi ticket to replace the builds, and that doesn't reset the current karma level to zero. The new builds may be completely broken. -- Fedora release 16 (Verne) - Linux 3.1.0-0.rc9.git0.0.fc16.x86_64 loadavg: 0.01 0.11 0.13 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Thu, 2011-10-20 at 00:30 +0200, Michael Schwendt wrote: On Wed, 19 Oct 2011 23:36:43 +0200, HA (Heiko) wrote: IMHO Rawhide should be the only place where version-control-snapshots of such an important component like glibc should be allowed. Maybe it would be better to let the value of positive karma depend on the severity of the package. That would mean that packages like glibc would require more positive karma for being pushed to stable than packages like gedit. Bodhi would need to be changed first. So far, what some package maintainers do is to wait for a first +1, then edit the bodhi ticket to replace the builds, and that doesn't reset the current karma level to zero. The new builds may be completely broken. I'm not aware of any case of a maintainer doing this intentionally, if that's what you're suggesting. Editing updates with new builds is a reasonably common action and there are perfectly legitimate reasons to do it. I thought the bug where Bodhi doesn't reset karma when you do this was supposed to be getting fixed... -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, Oct 19, 2011 at 5:51 PM, Rahul Sundaram methe...@gmail.com wrote: On 10/20/2011 01:06 AM, Adam Williamson wrote: On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice It is horrendously stupid to do this. It is not a question of being kind or nice. Fedora is not glibc's development platform. They should get their own and stop mucking around and breaking things unnecessarily and repeatedly. If they won't stop on their own, FESCo must push them to stop. Except that Fedora _has_ been glibc's development platform for as long as I can remember. The Fedora project might not think so, but it's exactly what upstream glibc does. They've made claims that they won't change things in glibc that depend on external items until it's in a Fedora release and they do their final upstream release just before Fedora does (much the chagrin of rel-eng since we've had to take a late upstream release that is essentially a rename after freeze). There are 3 possible solutions. I'll list them in what I think is order of preference. 1) Upstream changes their development model to follow the Fedora Branched method, introducing a development freeze upstream around the same time we branch. Non-trivial rewrites and/or features go into rawhide, bugfixes go into branched. 2) We ignore this still, and generally realize that yes the practice is somewhat dubious but for the most part a vastly large and insurmountable problem. Karma and testing have worked fairly well thus far. 3) We have a project hissy fit, stop accepting updates to the glibc package regardless of what upstream does based on a FESCo/FPC/Board edict. We eventually have a non-upstream glibc package maintainer that does all the work from item #1 while trying to not be at complete odds with upstream glibc. I'd vote for #1, but that's a much longer conversation that should be had upstream and before we even get close to bringing it to FESCo. josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: BEWARE: a problematic glibc made it to stable (F16)
On Wed, Oct 19, 2011 at 8:35 PM, Josh Boyer jwbo...@gmail.com wrote: On Wed, Oct 19, 2011 at 5:51 PM, Rahul Sundaram methe...@gmail.com wrote: On 10/20/2011 01:06 AM, Adam Williamson wrote: On Wed, 2011-10-19 at 15:30 -0400, Simo Sorce wrote: What did you downgrade to ? AFAIK Several people had to downgrade from -11 because of nsswitch issues ... seem glibc is not in good shape :-( You get to pick your breakage. If glibc maintainers would kindly stop pulling random git snapshots into a pending stable release that would be nice It is horrendously stupid to do this. It is not a question of being kind or nice. Fedora is not glibc's development platform. They should get their own and stop mucking around and breaking things unnecessarily and repeatedly. If they won't stop on their own, FESCo must push them to stop. Except that Fedora _has_ been glibc's development platform for as long as I can remember. The Fedora project might not think so, but it's exactly what upstream glibc does. They've made claims that they won't change things in glibc that depend on external items until it's in a Fedora release and they do their final upstream release just before Fedora does (much the chagrin of rel-eng since we've had to take a late upstream release that is essentially a rename after freeze). There are 3 possible solutions. I'll list them in what I think is order of preference. 1) Upstream changes their development model to follow the Fedora Branched method, introducing a development freeze upstream around the same time we branch. Non-trivial rewrites and/or features go into rawhide, bugfixes go into branched. 2) We ignore this still, and generally realize that yes the practice is somewhat dubious but for the most part a vastly large and That should read ... for the most part _isn't_ Sigh. josh -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
