[gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: rfc: escape sequences in logs)
Kent Fredric posted on Wed, 04 Sep 2013 23:38:40 +1200 as excerpted: > I see. I have a few gvim instances also reading/writing to that terminal > I didn't know about, interesting. Which brings up the privacy point. Anything getting this fancy and convoluted in terms of implementation is going to have to be very careful to narrow the scope of what it's looking at, lest it become the NSA in terms of dragnet. Even then, how many people would REALLY trust it, knowing the lengths to which it has to go to narrow down and focus in on JUST the info it needs? I don't think I'd trust it, for the same reason I don't trust the NSA doing similar things under "just trust us" pretenses. Would YOU? (Granted, our implementation would presumably be open, something the NSA most certainly is NOT, but still...) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
Dnia 2013-09-04, o godz. 22:19:30 Gilles Dartiguelongue napisał(a): > Le mercredi 04 septembre 2013 à 15:48 -0400, Ian Stakenvicius a écrit : > > On 04/09/13 03:44 PM, Gilles Dartiguelongue wrote: > > > Le mercredi 04 septembre 2013 à 15:23 -0400, Ian Stakenvicius a > > > écrit : [snip] > > >> > > >> By gdk-pixbuf.cache , you mean the 'loaders.cache' file that the > > >> eclass is now continuously updating? Which ebuild is going to > > >> 'own' it? > > > > > > yes, gdk-pixbuf is going to own it since it is the main loader > > > provider and the package that provides the tool to generate the > > > cache. > > > > > >> Also, is it owned by anything right now? IIRC we don't try > > >> particularly hard to support FEATURES="collision-protect" in the > > >> tree, but rather FEATURES="protect-owned", and so if the file is > > >> currently sitting there unowned by any package, afaik you > > >> shouldn't get any collisions by installing over it. > > > > > > it is not owned by any package right now but touching the file in > > > src_install made collision-protect abort the install. > > > > You had FEATURES="collision-protect" enabled" or the default > > FEATURES="protect-owned" ? > > the default, but since I only touched the file, maybe it's > FEATURES="config-protect-if-modified" kicking in ? The default is 'protect-owned' only. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 04/09/13 03:44 PM, Gilles Dartiguelongue wrote:
> Le mercredi 04 septembre 2013 à 15:23 -0400, Ian Stakenvicius a
> écrit : [snip]
>>
>> By gdk-pixbuf.cache , you mean the 'loaders.cache' file that the
>> eclass is now continuously updating? Which ebuild is going to
>> 'own' it?
>
> yes, gdk-pixbuf is going to own it since it is the main loader
> provider and the package that provides the tool to generate the
> cache.
>
>> Also, is it owned by anything right now? IIRC we don't try
>> particularly hard to support FEATURES="collision-protect" in the
>> tree, but rather FEATURES="protect-owned", and so if the file is
>> currently sitting there unowned by any package, afaik you
>> shouldn't get any collisions by installing over it.
>
> it is not owned by any package right now but touching the file in
> src_install made collision-protect abort the install.
You had FEATURES="collision-protect" enabled" or the default
FEATURES="protect-owned" ?
>
>> If you want to do that *and* maintain whatever is currently in
>> that file, you can use the trick sys-apps/openrc used to do: in
>> pkg_preinst, copy the system file (if it exists) into ${D}, and
>> then let that same copy be merged back into the system. Openrc
>> did it to get around CONFIG_PROTECT, but it had the unfortunate
>> side effect of making the package own the file. I don't know if
>> removal will be affected by this though if the contents of the
>> file change after the ebuild owning it was merged?
>
> That sounds like a good idea, I guess it does not cause a
> collision-protect error because the file is added to ${D} after
> comparison between ${D} and live file-system ?
>
No, it still does collide that first time if
FEATURES="collision-protect" is enabled. In fact, I do not believe
there is (by design) any way for this ebuild to 'take ownership' of a
file it doesn't already own without user intervention, if
FEATURES="collision-protect" is enabled.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
iF4EAREIAAYFAlInjogACgkQ2ugaI38ACPBzyQD9E6d71+zINTn6GWPPmOJHJL0I
K4IWNlanJJVE5WNpypkA/1bB1iYGQuVZIok1IssaGinme1FyJeUnDHy9PaXQTdTt
=sPlK
-END PGP SIGNATURE-
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 04/09/13 03:48 PM, Ian Stakenvicius wrote: > No, it still does collide that first time if > FEATURES="collision-protect" is enabled. In fact, I do not > believe there is (by design) any way for this ebuild to 'take > ownership' of a file it doesn't already own without user > intervention, if FEATURES="collision-protect" is enabled. > that is, the COLLISION_IGNORE option in the profile is the only way to do this. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) iF4EAREIAAYFAlInjs4ACgkQ2ugaI38ACPD3GwEAhRAvZRTLrDgP2q8rr2gkAPPA j4px+1xgBjsmDVa1JzoBAJsxHUPRnorS2Je76ulN9G1y5q89MoiB+PzzYFw1ibh/ =isxn -END PGP SIGNATURE-
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
Le mercredi 04 septembre 2013 à 15:23 -0400, Ian Stakenvicius a écrit :
> On 04/09/13 02:57 PM, Gilles Dartiguelongue wrote:
[snip]
> > Is there any other solution or is there any other point that would
> > move the balance from one solution to another ?
> >
> > This solution would also be applied to a couple of other commonly
> > regenerated files in Gnome ebuilds, like gtk-icon-cache, etc.
> >
>
> By gdk-pixbuf.cache , you mean the 'loaders.cache' file that the
> eclass is now continuously updating? Which ebuild is going to 'own' it?
yes, gdk-pixbuf is going to own it since it is the main loader provider
and the package that provides the tool to generate the cache.
> Also, is it owned by anything right now? IIRC we don't try
> particularly hard to support FEATURES="collision-protect" in the tree,
> but rather FEATURES="protect-owned", and so if the file is currently
> sitting there unowned by any package, afaik you shouldn't get any
> collisions by installing over it.
it is not owned by any package right now but touching the file in
src_install made collision-protect abort the install.
> If you want to do that *and* maintain whatever is currently in that
> file, you can use the trick sys-apps/openrc used to do: in
> pkg_preinst, copy the system file (if it exists) into ${D}, and then
> let that same copy be merged back into the system. Openrc did it to
> get around CONFIG_PROTECT, but it had the unfortunate side effect of
> making the package own the file. I don't know if removal will be
> affected by this though if the contents of the file change after the
> ebuild owning it was merged?
That sounds like a good idea, I guess it does not cause a
collision-protect error because the file is added to ${D} after
comparison between ${D} and live file-system ?
--
Gilles Dartiguelongue
Gentoo
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 04/09/13 04:19 PM, Gilles Dartiguelongue wrote: > Le mercredi 04 septembre 2013 à 15:48 -0400, Ian Stakenvicius a > écrit : >> >> You had FEATURES="collision-protect" enabled" or the default >> FEATURES="protect-owned" ? > > the default, but since I only touched the file, maybe it's > FEATURES="config-protect-if-modified" kicking in ? > Is the file stored in a CONFIG_PROTECT area? If that's the case you have much bigger headaches to deal with, i expect. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) iF4EAREIAAYFAlInlkgACgkQ2ugaI38ACPBlzAD/dzO8Nnd3XHVWnUh2kIE4ynF2 adAUNJcMBdghsHoM4MgA/RFRig4N9fExncCI782eUPdMfVNwKM0oVxo7zIuDgGmy =UaH3 -END PGP SIGNATURE-
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
Le mercredi 04 septembre 2013 à 15:48 -0400, Ian Stakenvicius a écrit : > On 04/09/13 03:44 PM, Gilles Dartiguelongue wrote: > > Le mercredi 04 septembre 2013 à 15:23 -0400, Ian Stakenvicius a > > écrit : [snip] > >> > >> By gdk-pixbuf.cache , you mean the 'loaders.cache' file that the > >> eclass is now continuously updating? Which ebuild is going to > >> 'own' it? > > > > yes, gdk-pixbuf is going to own it since it is the main loader > > provider and the package that provides the tool to generate the > > cache. > > > >> Also, is it owned by anything right now? IIRC we don't try > >> particularly hard to support FEATURES="collision-protect" in the > >> tree, but rather FEATURES="protect-owned", and so if the file is > >> currently sitting there unowned by any package, afaik you > >> shouldn't get any collisions by installing over it. > > > > it is not owned by any package right now but touching the file in > > src_install made collision-protect abort the install. > > You had FEATURES="collision-protect" enabled" or the default > FEATURES="protect-owned" ? the default, but since I only touched the file, maybe it's FEATURES="config-protect-if-modified" kicking in ? -- Gilles Dartiguelongue Gentoo signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
On 09/04/2013 12:44 PM, Gilles Dartiguelongue wrote:
> Le mercredi 04 septembre 2013 à 15:23 -0400, Ian Stakenvicius a écrit :
>> If you want to do that *and* maintain whatever is currently in that
>> file, you can use the trick sys-apps/openrc used to do: in
>> pkg_preinst, copy the system file (if it exists) into ${D}, and then
>> let that same copy be merged back into the system. Openrc did it to
>> get around CONFIG_PROTECT, but it had the unfortunate side effect of
>> making the package own the file. I don't know if removal will be
>> affected by this though if the contents of the file change after the
>> ebuild owning it was merged?
>
> That sounds like a good idea, I guess it does not cause a
> collision-protect error because the file is added to ${D} after
> comparison between ${D} and live file-system ?
Right. The collision check happens _before_ pkg_preinst, so it's
possible to create a file in ${D} during pkg_preinst in order to fool
collision-protect.
--
Thanks,
Zac
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
Dnia 2013-09-04, o godz. 20:57:41 Gilles Dartiguelongue napisał(a): > One last point to handle, how to migrate gdk-pixbuf.cache so that it is > owned by the ebuild ? > > I've discussed this with Michał and it seems two options are possible. > > 1. rm the file on the filesystem in pkg_preinst in gdk-pixbuf ebuild > pros: > - works immediately without fiddling with profiles (see 2) Except when it doesn't ;). > cons: > - no idea what PMS says about it, Michał told me it shouldn't work yet > my testing proves otherwise. PMS doesn't cover collision-protect at all. It's purely portage feature, and at the time I played with twisted portage still complained before pkg_preinst(). > 2. use COLLISION_IGNORE in profiles/base/make.conf > pros: > - does not leave the system without the cache file > cons: > - add a setting to base/make.conf for a long period of time to ensure > most of our user have migrated (how long would it be btw, 6 months, 1 > year ?) > - does not protect other packages from owning the package due to this > very solution for the time the setting is left in base/make.conf That's where you convince Zac to accept my patch that applies COLLISION_IGNORE only to unowned files. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 04/09/13 02:57 PM, Gilles Dartiguelongue wrote:
> One last point to handle, how to migrate gdk-pixbuf.cache so that
> it is owned by the ebuild ?
>
> I've discussed this with Michał and it seems two options are
> possible.
>
> 1. rm the file on the filesystem in pkg_preinst in gdk-pixbuf
> ebuild pros: - works immediately without fiddling with profiles
> (see 2) cons: - no idea what PMS says about it, Michał told me it
> shouldn't work yet my testing proves otherwise. - leaves the system
> with no loaders.cache for a while which could result in apps
> starting with no lots of missing icons.
>
> 2. use COLLISION_IGNORE in profiles/base/make.conf pros: - does not
> leave the system without the cache file cons: - add a setting to
> base/make.conf for a long period of time to ensure most of our user
> have migrated (how long would it be btw, 6 months, 1 year ?) - does
> not protect other packages from owning the package due to this very
> solution for the time the setting is left in base/make.conf
>
> 3. write a news item and let users handle it
>
>
> Is there any other solution or is there any other point that would
> move the balance from one solution to another ?
>
> This solution would also be applied to a couple of other commonly
> regenerated files in Gnome ebuilds, like gtk-icon-cache, etc.
>
By gdk-pixbuf.cache , you mean the 'loaders.cache' file that the
eclass is now continuously updating? Which ebuild is going to 'own' it?
Also, is it owned by anything right now? IIRC we don't try
particularly hard to support FEATURES="collision-protect" in the tree,
but rather FEATURES="protect-owned", and so if the file is currently
sitting there unowned by any package, afaik you shouldn't get any
collisions by installing over it.
If you want to do that *and* maintain whatever is currently in that
file, you can use the trick sys-apps/openrc used to do: in
pkg_preinst, copy the system file (if it exists) into ${D}, and then
let that same copy be merged back into the system. Openrc did it to
get around CONFIG_PROTECT, but it had the unfortunate side effect of
making the package own the file. I don't know if removal will be
affected by this though if the contents of the file change after the
ebuild owning it was merged?
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
iF4EAREIAAYFAlIniLAACgkQ2ugaI38ACPA1gwD/ZRbX/aD+S/SOMHrvPsKJ6S7/
0Dvyq7FDPGpVwPXPLIAA/iYpLP4SbLowsFqGWOhyNB6FdAS93TOmoy5MNKHddocc
=9Xqu
-END PGP SIGNATURE-
Re: [gentoo-dev] [PATCH] gnome2-utils.eclass add support for gdk-pixbuf cache update
One last point to handle, how to migrate gdk-pixbuf.cache so that it is owned by the ebuild ? I've discussed this with Michał and it seems two options are possible. 1. rm the file on the filesystem in pkg_preinst in gdk-pixbuf ebuild pros: - works immediately without fiddling with profiles (see 2) cons: - no idea what PMS says about it, Michał told me it shouldn't work yet my testing proves otherwise. - leaves the system with no loaders.cache for a while which could result in apps starting with no lots of missing icons. 2. use COLLISION_IGNORE in profiles/base/make.conf pros: - does not leave the system without the cache file cons: - add a setting to base/make.conf for a long period of time to ensure most of our user have migrated (how long would it be btw, 6 months, 1 year ?) - does not protect other packages from owning the package due to this very solution for the time the setting is left in base/make.conf 3. write a news item and let users handle it Is there any other solution or is there any other point that would move the balance from one solution to another ? This solution would also be applied to a couple of other commonly regenerated files in Gnome ebuilds, like gtk-icon-cache, etc. -- Gilles Dartiguelongue Gentoo
Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in net-dialup/sendpage: sendpage-1.1.0-r2.ebuild ChangeLog sendpage-1.1.0-r1.ebuild
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 04/09/13 01:28 AM, Sergey Popov wrote:
> 02.09.2013 19:29, Ian Delaney (idella4) пишет:
>> idella4 13/09/02 15:29:57
>>
>> Modified: ChangeLog Added:
>> sendpage-1.1.0-r2.ebuild Removed:
>> sendpage-1.1.0-r1.ebuild Log: revbump -> EAPI 5, remove old
>>
>> (Portage version: 2.2.0/cvs/Linux x86_64, signed Manifest commit
>> with key 0xB8072B0D)
>>
>> Revision ChangesPath 1.13
>> net-dialup/sendpage/ChangeLog
>>
>> file :
>> http://sources.gentoo.org/viewvc.cgi/gentoo-x86/net-dialup/sendpage/ChangeLog?rev=1.13&view=markup
>>
>>
plain:
http://sources.gentoo.org/viewvc.cgi/gentoo-x86/net-dialup/sendpage/ChangeLog?rev=1.13&content-type=text/plain
>> diff :
>> http://sources.gentoo.org/viewvc.cgi/gentoo-x86/net-dialup/sendpage/ChangeLog?r1=1.12&r2=1.13
>>
>>
>>
Index: ChangeLog
>> ===
>>
>>
RCS file: /var/cvsroot/gentoo-x86/net-dialup/sendpage/ChangeLog,v
>> retrieving revision 1.12 retrieving revision 1.13 diff -u -r1.12
>> -r1.13 --- ChangeLog 14 Jun 2012 01:50:05 - 1.12 +++
>> ChangeLog2 Sep 2013 15:29:57 - 1.13 @@ -1,6 +1,12 @@ #
>> ChangeLog for net-dialup/sendpage -# Copyright 2000-2012 Gentoo
>> Foundation; Distributed under the GPL v2 -# $Header:
>> /var/cvsroot/gentoo-x86/net-dialup/sendpage/ChangeLog,v 1.12
>> 2012/06/14 01:50:05 zmedico Exp $ +# Copyright 2000-2013 Gentoo
>> Foundation; Distributed under the GPL v2 +# $Header:
>> /var/cvsroot/gentoo-x86/net-dialup/sendpage/ChangeLog,v 1.13
>> 2013/09/02 15:29:57 idella4 Exp $ + +*sendpage-1.1.0-r2 (02 Sep
>> 2013) + + 02 Sep 2013; Ian Delaney
>> +sendpage-1.1.0-r2.ebuild, + -sendpage-1.1.0-r1.ebuild: +
>> revbump -> EAPI 5, remove old
>>
>> 14 Jun 2012; Zac Medico
>> sendpage-1.1.0-r1.ebuild: inherit user for enewgroup and
>> enewuser
>>
>>
>>
>> 1.1
>> net-dialup/sendpage/sendpage-1.1.0-r2.ebuild
>>
>> file :
>> http://sources.gentoo.org/viewvc.cgi/gentoo-x86/net-dialup/sendpage/sendpage-1.1.0-r2.ebuild?rev=1.1&view=markup
>>
>>
plain:
http://sources.gentoo.org/viewvc.cgi/gentoo-x86/net-dialup/sendpage/sendpage-1.1.0-r2.ebuild?rev=1.1&content-type=text/plain
>>
>> Index: sendpage-1.1.0-r2.ebuild
>> ===
>>
>>
# Copyright 1999-2013 Gentoo Foundation
>> # Distributed under the terms of the GNU General Public License
>> v2 # $Header:
>> /var/cvsroot/gentoo-x86/net-dialup/sendpage/sendpage-1.1.0-r2.ebuild,v
>> 1.1 2013/09/02 15:29:57 idella4 Exp $
>>
>> EAPI=5
>>
>> inherit perl-module eutils user
>>
>> MY_P=${PN}-1.001 DESCRIPTION="Dialup alphapaging software."
>> HOMEPAGE="http://www.sendpage.org/";
>> SRC_URI="http://www.sendpage.org/download/${MY_P}.tar.gz";
>> S="${WORKDIR}/${MY_P}"
>>
>> LICENSE="GPL-2" SLOT="0" KEYWORDS="~amd64 ~ppc ~x86" # This
>> package warrants IUSE doc IUSE=""
>>
>> DEPEND="!net-misc/hylafax >=dev-perl/Device-SerialPort-0.13
>> >=dev-perl/MailTools-1.44 >=virtual/perl-libnet-1.11
>> >=dev-perl/Net-SNPP-1.13 dev-perl/DBI"
>>
>> mydoc="FEATURES email2page.conf sendpage.cf snpp.conf"
>>
>> pkg_setup() { enewgroup sms enewuser sendpage -1 -1
>> /var/spool/sendpage sms }
>>
>> PATCHES=( "${FILESDIR}"/${PV}-makefile.patch )
>>
>> src_install() { perl-module_src_install insinto /etc doins
>> sendpage.cf newinitd "${FILESDIR}"/sendpage.initd sendpage
>> diropts -o sendpage -g sms -m0770 keepdir /var/spool/sendpage #
>> Separate docs/ content from ${mydoc[@]} docompress -x
>> /usr/share/doc/${PF}/text/ docinto text/ dodoc docs/* }
>>
>>
>>
>>
>
> EAPI=5 has no implicit RDEPEND="${DEPEND}". Does this package
> really has no run-time dependendies?
>
>
Well, perl-module.eclass adds an RDEPEND of dev-lang/perl , so there's
that. If it's just perl scripts, though (and I haven't checked if it
is or not), then that probably would be the only hard RDEPEND...
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
iF4EAREIAAYFAlInPUwACgkQ2ugaI38ACPDxOwEAm+pPqZUyvJUe9jL10VetnKMJ
A07EXClczp7QPnwPE/8A/As91QSuriHIpULiFaddqCBE1Bm3eBgS2dH0vobIf3nD
=xbZU
-END PGP SIGNATURE-
Re: [gentoo-dev] RFC: final version of git-r3 (+ compat for git-2)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 03/09/13 08:19 PM, William Hubbs wrote: > On Tue, Sep 03, 2013 at 01:37:25PM +0200, Michał Górny wrote: >> Dnia 2013-09-03, o godz. 12:24:49 Markos Chandras >> napisał(a): >> >>> On 3 September 2013 12:17, Michał Górny >>> wrote: Dnia 2013-09-03, o godz. 11:53:22 Markos Chandras napisał(a): > On 3 September 2013 11:45, Michał Górny > wrote: >> Hello, all. >> >> >> I'm attaching git-r3.eclass and a patch to git-2.eclass >> that adds ability to use git-r3 internally via make.conf >> switch. > > I am a bit skeptical about this. Why would someone want to > do this apart from testing the git-r3 eclass without > touching the existing git-r2 compatible ebuilds? And why do > you want to do that in the first place? If the maintainer > is happy with how git-r2 works with his ebuilds I see no > reason to allow users to silently override that eclass. The goal is to give git-r3 most testing it could get before it gets widely used. I'd prefer catching corner cases sooner than later. -- Best regards, Michał Górny >>> >>> Ok but I don't think allowing users to override eclasses like >>> this is a good thing. Maintainers expect the existing ebuilds >>> to work with git-r2. If they start getting bugs because a user >>> silently overrode the eclass the this is not going to be >>> pleasant. >> >> It is not done silently. It comes with big fat warning at the >> top: >> >> + ewarn "Using git-r3 backend in git-2. Not everything is >> supported." + ewarn "Expect random failures and have fun >> testing." > > I tend also to want to err on the side of caution here. I don't > think users should be able to change something in make.conf that > affects which eclass an ebuild uses. > > I would suggest putting a warning in the git-2.eclass that starts > encouraging maintainers to migrate their ebuilds to git-r3. > To be honest, I see no problems with the option to source git-3 within git-2 -- it's for testing, it's internal, it's not meant to be supportable, and it's a precursor to official git-3 adoption and deprecation of git-2. There is a warning already that end-users will see (as will we, in build.log's if bugs are filed), and I don't think it's any more dangerous than say, setting 'I_KNOW_WHAT_IM_DOING="yes"' in make.conf. If this were to be an option that was explicitly meant for end-users to use on a regular basis, then I would agree that it's not proper. But this isn't some sort of official new feature, it's just a workaround to allow something to get more testing. I would be quite surprised if anyone outside of ebuild developers/maintainers (whether they be dev's or otherwise) would enable this option, or even know it exists (outside of those that read this ML, of course) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) iF4EAREIAAYFAlInPKQACgkQ2ugaI38ACPDrrgD+MyynAyYF4u9WhH/eAn2XT26P OZIfSpVSdVB7/fdeqcEA+wUJgThucM5pZdf3QY8g2T15GA6McED+Hc/iABQG85Gk =Tp+l -END PGP SIGNATURE-
[gentoo-dev] Re: rfc: escape sequences in logs
William Hubbs writes: > On Mon, Sep 02, 2013 at 09:41:28PM +0200, Michał Górny wrote: >> That is a bug in pybugz and not an argument, you know. > > I said "things like pybugz". > > Bugzilla allowing control characters in the xml is the issue. The python > xmlrpc library raises an exception for malformed xml because of it, so > there isn't much pybugz, or anything that uses that library, can do > about it. Right. The bug is in bugzilla itself, not pybugz. I did some research. Escape and all of the other non-whitespace ASCII control characters are illegal in XML. It is also not valid to escape them with entities, like . However, Bugzilla's XML-RPC interface sends them anyway. This is only a problem for build logs included inline in bug comments. Attached logs are fine, with or without the color codes. Maybe the codes could be stripped by pybugz before processing the XML? -- Chris
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
Dnia 2013-09-04, o godz. 23:45:44 Kent Fredric napisał(a): > On 4 September 2013 21:59, Michał Górny wrote: > > > And how are you going to implement this? I doubt that fd/vt input has > > any sort of 'writing process id' indicator. > > > > > Though granted, my other post is not going to be useful on a line-by-line > basis. > > The obvious easy approach is have an exec launcher of some kind that > launches processes with IO redirected to an intermediary process that > simply prepends arbitrary strings to every line sent to them, prior to > those processes piping the data to the final aggregate output. > > But this would require large amounts of pipe work and fork+ipc magic. > > Its surely doable, I recall seeing parts of paludis that do stuff like this. > > Just those pipes are inherently easy to break in my experience. That's my point. There's no simple way of doing such a thing, and any 'hard' way involves a lot of fragile hacks and assumptions. And more of that, more likely it will actually break something that relies on at least some sanity of host. I can't really imagine going anywhere without another LD_PRELOAD library. Then, I don't know if platforms other than Linux have any tools for proper chaining of LD_PRELOAD (think of sandbox). That's just one issue. Then, think of pipes. Some programs use stdin/stdout to communicate, maybe even stderr. You need to know which handles you can actually touch and which not. That should be fairly easy to achieve but still I suspect it would be fragile. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
On 4 September 2013 21:59, Michał Górny wrote: > And how are you going to implement this? I doubt that fd/vt input has > any sort of 'writing process id' indicator. > Though granted, my other post is not going to be useful on a line-by-line basis. The obvious easy approach is have an exec launcher of some kind that launches processes with IO redirected to an intermediary process that simply prepends arbitrary strings to every line sent to them, prior to those processes piping the data to the final aggregate output. But this would require large amounts of pipe work and fork+ipc magic. Its surely doable, I recall seeing parts of paludis that do stuff like this. Just those pipes are inherently easy to break in my experience. -- Kent
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
On 4 September 2013 21:59, Michał Górny wrote: > And how are you going to implement this? I doubt that fd/vt input has > any sort of 'writing process id' indicator. > In one terminal: cat -vET In another: pgrep -x cat # 199935 ls -la /proc/199935/fd/ dr-x-- 2 kent kent 0 Sep 4 23:29 . dr-xr-xr-x 8 kent kent 0 Sep 4 23:28 .. lrwx-- 1 kent kent 64 Sep 4 23:29 0 -> /dev/pts/3 lrwx-- 1 kent kent 64 Sep 4 23:29 1 -> /dev/pts/3 lrwx-- 1 kent kent 64 Sep 4 23:29 2 -> /dev/pts/3 So you can certainly get the information the ohter way round. now... Term 1: $ tty /dev/pts/3 $ cat -vET Term 2: for i in /proc/[0-9]*; do cmd=$(cat $i/comm); for j in $i/fd/*; do echo -n "$cmd:$j:"; readlink $j ; echo ; done done | grep "pts/3" bash:/proc/159261/fd/0:/dev/pts/3 bash:/proc/159261/fd/1:/dev/pts/3 bash:/proc/159261/fd/2:/dev/pts/3 bash:/proc/159261/fd/255:/dev/pts/3 gvim:/proc/175475/fd/0:/dev/pts/3 gvim:/proc/175475/fd/1:/dev/pts/3 gvim:/proc/175475/fd/2:/dev/pts/3 gvim:/proc/175642/fd/0:/dev/pts/3 gvim:/proc/175642/fd/1:/dev/pts/3 gvim:/proc/175642/fd/2:/dev/pts/3 gvim:/proc/199494/fd/0:/dev/pts/3 gvim:/proc/199494/fd/1:/dev/pts/3 gvim:/proc/199494/fd/2:/dev/pts/3 cat:/proc/207567/fd/0:/dev/pts/3 cat:/proc/207567/fd/1:/dev/pts/3 cat:/proc/207567/fd/2:/dev/pts/3 I see. I have a few gvim instances also reading/writing to that terminal I didn't know about, interesting. -- Kent
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
On Wed, 4 Sep 2013 11:59:37 +0200 Michał Górny wrote: > And how are you going to implement this? I doubt that fd/vt input has > any sort of 'writing process id' indicator. Yeah, will require some inspection into how this works and what information we have available; if that indicator isn't present and upstream doesn't want to add it, we can still opt to hook into the write calls and make the information available that way. I can't imagine there is completely no way to do this; a first example, strace, comes to mind that can do this. So a stripped down form of that would be one way to do it. Assuming we can't get direct information. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
Dnia 2013-09-04, o godz. 11:24:22 Tom Wijsman napisał(a): > On Wed, 4 Sep 2013 09:17:11 +0200 > Michał Górny wrote: > > > Dnia 2013-09-03, o godz. 18:57:12 > > "Walter Dnes" napisał(a): > > > > > On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote > > > > > > > That is not what this is about, this is about having escape > > > > sequences in build logs obtained from Bugzilla; because, they aid > > > > in skimming through logs (until we implement the feature I asked > > > > for in subject). > > > > > > "The road to binary syslog files is paved with good intentions", > > > or something like that. Question... why does it have to be escape > > > sequences? Can't it be readable plain text? E.g. something like... > > > > > > //STDERR.OUT.START > > > foo > > > bar > > > blah blah blah > > > //STDERR.OUT.END > > > > > > This would be easy to grep and/or parse in bash. > > > > Especially if one is interspersed with the other. I suggest trying > > first, then suggesting it to others. > > Definitely do not want them on their own line; instead something like > > OUT:make:1000: ... > ERR:gcc:1001: ... > ERR:gcc:1001: ... > ERR:gcc:1001: ... > ERR:gcc:1001: ... > ERR:make:1000: *** [...] Error 1 > ERR:make:1000: make[2]: *** Waiting for unfinished jobs > > If you then grep the last non-make and non-portage STDERR, you simply > get just the gcc lines you actually want. From there you can grep the > lines around it for context. And how are you going to implement this? I doubt that fd/vt input has any sort of 'writing process id' indicator. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
On Wed, 4 Sep 2013 09:17:11 +0200 Michał Górny wrote: > Dnia 2013-09-03, o godz. 18:57:12 > "Walter Dnes" napisał(a): > > > On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote > > > > > That is not what this is about, this is about having escape > > > sequences in build logs obtained from Bugzilla; because, they aid > > > in skimming through logs (until we implement the feature I asked > > > for in subject). > > > > "The road to binary syslog files is paved with good intentions", > > or something like that. Question... why does it have to be escape > > sequences? Can't it be readable plain text? E.g. something like... > > > > //STDERR.OUT.START > > foo > > bar > > blah blah blah > > //STDERR.OUT.END > > > > This would be easy to grep and/or parse in bash. > > Especially if one is interspersed with the other. I suggest trying > first, then suggesting it to others. Definitely do not want them on their own line; instead something like OUT:make:1000: ... ERR:gcc:1001: ... ERR:gcc:1001: ... ERR:gcc:1001: ... ERR:gcc:1001: ... ERR:make:1000: *** [...] Error 1 ERR:make:1000: make[2]: *** Waiting for unfinished jobs If you then grep the last non-make and non-portage STDERR, you simply get just the gcc lines you actually want. From there you can grep the lines around it for context. You could then have a tool that tells you something like ... === ERROR at build.log:123 === ... ... ... ... === /ERROR at build.log:123 === *** [...] Error 1 make[2]: *** Waiting for unfinished jobs and that allows you to immediately see the error right away; additionally, it shows a few lines of context before and after. If you need more context, the line number allows you to jump right to it in the build.log if you need even more context. We could even build this feature into Bugzilla... -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
On Wed, 4 Sep 2013 18:03:14 +1200 Kent Fredric wrote: > On 4 September 2013 08:11, Tom Wijsman wrote: > > > And then I asked the questions that I'd like to see answered: > > > > > > Why do they not belong there? What do people have to do who want > > them? > > > > If anyone needs a poster child for the sort of escape sequence > outputs that most definitely are of no value to somebody reading the > log after the fact, the output from vims' test suite is a good > example. > > Granted, it uses a very wide variety of terminal escape codes, which > include, but are not limited to, window resizing control characters. But how much of these are actually relevant to the majority of the build logs that we receive; do they even contain those characters, or are they actually already stripped? I haven't seen any build log do window resizing so far for example. > I think given that context, it may be sane to restrict log control > characters to a specific subset of control characters, specifically > basic colour/highlighting control characters, which are at least > somewhat standardized and not too device specific [... SNIP ...] +1 The idea of just having the useful subset sounds really nice. > Either way, if you were to introduce such a variable, you could have a > standard defined value that Gentoo agree upon, which I'd imagine > might be "none erasures:apply" or something like that, and end users > can turn them back on if they want, with a caveat that users should > apply a standard stripping to these logs before submitting them to > bugzilla, with a tool that comes standard with portage to facilitate > stripping logs for submission to bugzilla. -1 I'd rather not make submission more complex than it needs to be; if we want any processing to happen here, we should probably let Bugzilla do this to keep things easy for the user. > This gives users the control they want, the features they want if > they need them, and still gives gentoo staff an easier time if it is > determined escape codes are not useful for bug reports. The hard part will be the determination. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?
On Wed, 4 Sep 2013 06:25:14 + (UTC) Duncan <1i5t5.dun...@cox.net> wrote: > Tom Wijsman posted on Tue, 03 Sep 2013 23:16:11 +0200 as excerpted: > > > Currently the logs aren't > > search and grep compatible because you have no indication where the > > last error is and which process has output that > > Quite apart from the ansi-color discussion, I've had reasonable luck > simply searching/grepping for "error" here. Sure, a lot of packages > build *error* files of some sort and they're normally > false-positives, but it does cut down the search space considerably, > and there's usually only a couple such false-positives to worry about > per-package. When you have to go through tenths to hundreds of those on a daily base, those "couple such false-positives" add up a lot; considering that stderr indication will get the exact line you actually need. Directly seeing the error (or even have it on my clipboard, 'cause I am a bug wrangler whom places it in the summary) is much faster than having to scroll through the output looking for the right line (and still having to select and copy it). In terms of what I need to process, we talk about changing O(n) to O(1). Oh, then you should know, it's not always "error" it says; other times, for example when failing tests, it says "fail"; and when you run some other generation tool (doc, ...) you see something different, sometimes it is an exception you get to see. Those are all easy to exactly match with [see subject], but not at all exact with our current logs. Rather than that we bike shed over something so unimportant as escape codes; I'd rather see our build logs improve with necessary information, in a way we can more efficiently deal with them than having to do the matching completely manually ourselves. It is in our philosophy to "to design tools and systems that allow a user to do that work as pleasantly and efficiently as possible, as they see fit." (user is in this case Gentoo Developer) so I don't see why we would keep dealing with build log unpleasant an inefficient. http://www.gentoo.org/main/en/philosophy.xml -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs?
On Tue, 3 Sep 2013 21:43:24 -0400 "Walter Dnes" wrote: > On Tue, Sep 03, 2013 at 11:44:45PM +0200, Tom Wijsman wrote > > > +1 I am still not convinced we are experiencing an actual practical > > problem for the majority of the build logs that are attached; we've > > been doing this for years, why is it so suddenly considered a > > problem? > > As I pointed out in a message to the previous thread... > > == > WARN: prepare > It seems that you need to set USE_PYTHON to make sure that legacy > packages will be built with respect to PYTHON_TARGETS correctly: > > USE_PYTHON='.[35;1m2.7.[0m' > > == > > See https://bugs.gentoo.org/show_bug.cgi?id=463954 That's an easy one, it is clearly 2.7 surrounded by escape codes; but that is not something the maintainer that downloads the build.log from Bugzilla needs to know. I don't really see this as a counter example. The user won't see this at the end of the output or in elogv; and even when catting it to his TERM (which is the original one almost always) it will also show colored. So, I don't see the problem here. But let me restate, this discussion is _not_ about our users; we are discussing the build logs as downloaded from Bugzilla, users _already_ have the means to disable (in make.conf) or strip it (we discussed many examples) on their own if they don't want to use an appropriate viewer. -- With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : tom...@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: final version of git-r3 (+ compat for git-2)
Dnia 2013-09-03, o godz. 19:53:22 James Cloos napisał(a): > (I think I forgot to mention when I wrote about keeping git-2 around for > a while that I like the plan for git-3; that should have been explicit.) > > It looks good. > > I haven't worked out what the storage names under EGIT3_STORE_DIR will > be for submodules, though. If multiple parent gits want the same > submodule, and/or if a package exists for it in its own right, will > the same store clone be used for all of them? Including when some use > git://, others https://, et al? Yes, that's the goal. https://bitbucket.org/mgorny/flaggie.git -> mgorny_flaggie.git git://git.overlays.gentoo.org/dev/aballier.git https://git.overlays.gentoo.org/gitroot/dev/aballier.git -> both to dev_aballier.git (gitroot/ is stripped) The pattern magic can be easily extended if necessary. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: Can we have process names and stdout / stderr indication to more efficiently parse build logs? (was: Re: [gentoo-dev] rfc: escape sequences in logs)
Dnia 2013-09-03, o godz. 18:57:12 "Walter Dnes" napisał(a): > On Tue, Sep 03, 2013 at 10:15:39PM +0200, Tom Wijsman wrote > > > That is not what this is about, this is about having escape sequences > > in build logs obtained from Bugzilla; because, they aid in skimming > > through logs (until we implement the feature I asked for in subject). > > "The road to binary syslog files is paved with good intentions", or > something like that. Question... why does it have to be escape > sequences? Can't it be readable plain text? E.g. something like... > > //STDERR.OUT.START > foo > bar > blah blah blah > //STDERR.OUT.END > > This would be easy to grep and/or parse in bash. Especially if one is interspersed with the other. I suggest trying first, then suggesting it to others. -- Best regards, Michał Górny signature.asc Description: PGP signature
