Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
On 04-12-2021 10:24:23 +0100, Michał Górny wrote:
> On Sat, 2021-12-04 at 10:15 +0100, Fabian Groffen wrote:
> > On 04-12-2021 10:13:09 +0100, Michał Górny wrote:
> > > On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote:
> > > > Why don't you change your color.map instead of changing the default for
> > > > everyone?
> > > 
> > > Why should we keep a stupid default?  Should we optimize Gentoo for
> > > people who don't want to be able to read Portage's output?
> > 
> > You're assuming everyone uses the terminal in the way you do.  I simply
> > don't think that's how the world looks like.
> 
> On the other hand, you're assuming that everyone uses the terminal
> in the way you do.

It eludes me how you came to that conclusion.

> > No need for calling things stupid, IMO.
> 
> Using dark blue on black background is stupid.

... then don't use black background or dark blue text?

Now, if you would make a supported claim that all terminals we install
use a black background by default, your change becomes more valid.

However, we then still don't know if people leave that default or use
something else, but we could make some educated guess about the amount
of people not changing the default.

My point, because I think this wasn't clear to you, is and always was,
how many people is this change going to be disruptive to.  And should
we make a hint to users when they install this version of Portage that
they can revert/change this by altering color.map (and how)?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
On 04-12-2021 10:13:09 +0100, Michał Górny wrote:
> On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote:
> > Why don't you change your color.map instead of changing the default for
> > everyone?
> 
> Why should we keep a stupid default?  Should we optimize Gentoo for
> people who don't want to be able to read Portage's output?

You're assuming everyone uses the terminal in the way you do.  I simply
don't think that's how the world looks like.

No need for calling things stupid, IMO.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
Why don't you change your color.map instead of changing the default for
everyone?

Thanks,
Fabian

On 04-12-2021 05:48:49 +0100, Michał Górny wrote:
> The "darkblue" color is often barely visible on dark terminals which
> makes reading emerge output really hard (I basically have to copy-paste
> it a lot in order to be able to read it at all).  Replace it with teal
> that does not seem to have any significant use in the output.
> 
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/output.py | 6 +++---
>  man/color.map.5   | 6 +++---
>  2 files changed, 6 insertions(+), 6 deletions(-)
> 
> 
> The before-after screenshot can be found on the PR:
> https://github.com/gentoo/portage/pull/776
> 
> 
> diff --git a/lib/portage/output.py b/lib/portage/output.py
> index 42f487f8a..77375b012 100644
> --- a/lib/portage/output.py
> +++ b/lib/portage/output.py
> @@ -157,7 +157,7 @@ _styles["UNMERGE_WARN"] = ("red",)
>  _styles["SECURITY_WARN"] = ("red",)
>  _styles["MERGE_LIST_PROGRESS"] = ("yellow",)
>  _styles["PKG_BLOCKER"] = ("red",)
> -_styles["PKG_BLOCKER_SATISFIED"] = ("darkblue",)
> +_styles["PKG_BLOCKER_SATISFIED"] = ("teal",)
>  _styles["PKG_MERGE"] = ("darkgreen",)
>  _styles["PKG_MERGE_SYSTEM"] = ("darkgreen",)
>  _styles["PKG_MERGE_WORLD"] = ("green",)
> @@ -165,8 +165,8 @@ _styles["PKG_BINARY_MERGE"] = ("purple",)
>  _styles["PKG_BINARY_MERGE_SYSTEM"] = ("purple",)
>  _styles["PKG_BINARY_MERGE_WORLD"] = ("fuchsia",)
>  _styles["PKG_UNINSTALL"] = ("red",)
> -_styles["PKG_NOMERGE"] = ("darkblue",)
> -_styles["PKG_NOMERGE_SYSTEM"] = ("darkblue",)
> +_styles["PKG_NOMERGE"] = ("teal",)
> +_styles["PKG_NOMERGE_SYSTEM"] = ("teal",)
>  _styles["PKG_NOMERGE_WORLD"] = ("blue",)
>  _styles["PROMPT_CHOICE_DEFAULT"] = ("green",)
>  _styles["PROMPT_CHOICE_OTHER"] = ("red",)
> diff --git a/man/color.map.5 b/man/color.map.5
> index 288bf7fbd..92a1baa91 100644
> --- a/man/color.map.5
> +++ b/man/color.map.5
> @@ -40,7 +40,7 @@ Defines color used for numbers indicating merge progress.
>  \fBPKG_BLOCKER\fR = \fI"red"\fR
>  Defines color used for unsatisfied blockers.
>  .TP
> -\fBPKG_BLOCKER_SATISFIED\fR = \fI"darkblue"\fR
> +\fBPKG_BLOCKER_SATISFIED\fR = \fI"teal"\fR
>  Defines color used for satisfied blockers.
>  .TP
>  \fBPKG_MERGE\fR = \fI"darkgreen"\fR
> @@ -63,10 +63,10 @@ package.
>  Defines color used for world packages planned to be merged using a binary
>  package.
>  .TP
> -\fBPKG_NOMERGE\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE\fR = \fI"teal"\fR
>  Defines color used for packages not planned to be merged.
>  .TP
> -\fBPKG_NOMERGE_SYSTEM\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE_SYSTEM\fR = \fI"teal"\fR
>  Defines color used for system packages not planned to be merged.
>  .TP
>  \fBPKG_NOMERGE_WORLD\fR = \fI"blue"\fR
> -- 
> 2.34.1
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-05 Thread Fabian Groffen
In all fairness, there haven't been that much incidents with Portage in
the past under Zac's supervision, isn't it a bit overkill to
bureaucratise the release model just after one incident?  It appears to
me that changes to Portage need to be considered very carefully, always,
since it affects everyone.

Thanks,
Fabian

On 05-10-2021 10:31:40 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> I've been thinking about this for some time already, and the recent
> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
> branch of Portage.
> 
> Roughly, the idea is that:
> 
> - master becomes 3.1.x, and primary development happens there
> 
> - 3.0.x becomes the LTS branch and only major bugfixes are backported
> there
> 
> As things settle down in the future, master would become 3.2.x, 3.1.x
> would become LTS, 3.0.x will be discontinued and so on.
> 
> WDYT?
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-10-05 Thread Fabian Groffen
On 05-10-2021 09:35:32 +0200, Michał Górny wrote:
> On Thu, 2021-09-30 at 09:18 +0200, Fabian Groffen wrote:
> > > > Final question, am I understanding correctly that normal lines are not
> > > > prefixed with something?  Would it be, for consistency, alignment, and
> > > > certainty of selecting rows something to use a prefix for those lines
> > > > too (assuming they aren't at this point)?
> > > 
> > > I don't know, we've never done that.  I suppose it would be possible but
> > > it is even more controversial and unlike the proposed changes, it would
> > > actually require mangling the process output.
> > 
> > If I remember correctly, Portage already does.  In which case, doing
> > this (even if it were adding leading spaces) would not be that much
> > work?
> > 
> 
> I'm afraid this is not that simple.  You need to account for all escape
> sequences that can affect editing already output data including clean
> handling of line wrapping.  In the end, we'd have to have a complete
> detachtty-class terminal emulator inside Portage.

Fair enough, thanks for looking into it.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-10-03 Thread Fabian Groffen
On 02-10-2021 23:03:56 -0400, Ionen Wolkens wrote:
> On Sat, Oct 02, 2021 at 10:53:37PM -0400, Ionen Wolkens wrote:
> > Guess there's a lot of other options that could be considered as well.
> > 
> > --- files
> > >>> text
> >  * current, it wasn't aligned now that I look at it again
> > (relying only on color to convey type feels clearly wrong to me)
> > 
> > --- files
> > >>>> text
> > [QA] new based on current PR
> > 
> > >>> text
> > [QA] aligned 1 character further, maybe skipping changing >>> is fine?
> > (then again that it's further is what threw me off at first)
> > 
> > >>> text
> > QA* similar to before, but aligned using only 3 chars
> > 
> > >>> text
> > [Q] kinda more obscure but it can work
> > 
> 
> Guess should also add these:
> >>> text
> Q* Notice:
> E* Some error happened
> (closest to before by making use of the former leading space, thus
>  no alignment changes)

FWIW, I like this one.  Perhaps even with lowercase

make[4]: leaving directory src
q* soname lacks version
e* failed to die

Fabian

> 
> >>> text
> QA Notice:
> EE Some error happened
> (at least clearer than Q* Notice, but unsure about no separator.. guess
> it could work?)
> 
> > >>>> text
> > QA* probably closest to how it was before alignment-wise, but meh at 4 >
> > 
> > >>> message
> > QA> not convinced about this one, but throwing it here anyway
> > (other characters could be considered as well)
> > 
> > Maybe a poll of some kind may help, personally undecided on what I
> > like better beside agreeing that a change is needed.
> 
> 
> -- 
> ionen



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-09-30 Thread Fabian Groffen
On 30-09-2021 08:44:33 +0200, Michał Górny wrote:
> On Thu, 2021-09-30 at 08:40 +0200, Fabian Groffen wrote:
> > Hi,
> > 
> > Would it be possible to have some switch (e.g. --style=legacy) that
> > controls this new vs. the old behaviour?  Would perhaps allow
> > applications that parse the output to work via setting this in the
> > global opts.
> 
> Patches welcome.  It shouldn't be hard, my commit shows which files need
> to be edited to alter the prefixes and how to pass them into ebd.

I see.

> > In addition, much like the colour map, how do you see this change in
> > light of eclasses, init-scripts, etc. that also use the same scheme as
> > Portage at the moment?  Would you expect to change those too at some
> > point?
> 
> Eclasses are supposed to use standard einfo, elog... functions, so they
> should just work™.  If someone's reinventing the wheel, it's not my
> problem.
> 
> Init scripts aren't supposed to be used inside the PM, so that's out of
> scope.

I was just referring to the overall "feel" of Gentoo, which your work
changes.  It is ok that you don't plan on doing anything there.

> > Final question, am I understanding correctly that normal lines are not
> > prefixed with something?  Would it be, for consistency, alignment, and
> > certainty of selecting rows something to use a prefix for those lines
> > too (assuming they aren't at this point)?
> 
> I don't know, we've never done that.  I suppose it would be possible but
> it is even more controversial and unlike the proposed changes, it would
> actually require mangling the process output.

If I remember correctly, Portage already does.  In which case, doing
this (even if it were adding leading spaces) would not be that much
work?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-09-30 Thread Fabian Groffen
Hi,

Would it be possible to have some switch (e.g. --style=legacy) that
controls this new vs. the old behaviour?  Would perhaps allow
applications that parse the output to work via setting this in the
global opts.

In addition, much like the colour map, how do you see this change in
light of eclasses, init-scripts, etc. that also use the same scheme as
Portage at the moment?  Would you expect to change those too at some
point?

Final question, am I understanding correctly that normal lines are not
prefixed with something?  Would it be, for consistency, alignment, and
certainty of selecting rows something to use a prefix for those lines
too (assuming they aren't at this point)?

Thanks,
Fabian

On 28-09-2021 17:36:25 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> I know I'm going to regret asking this... but I've prepared a change to
> the Portage output format and I think it asks for a wider discussion
> than internally in Portage team.
> 
> The primary problem with the current output format is that different
> kinds of messages differ only in color.  This makes them
> indistinguishable without colors and hard to grep.  ago's been asking
> for a better way to grep for QA warnings and this is pretty much what
> motivated me to do this.
> 
> The proposed new format distinguished message types both using colors
> and strings.  This is roughly inspired by Xorg logs.  For example,
> instead of:
> 
>  * some message
>  * other message
>  * hell if i know what this is
> 
> You get:
> 
> [WW] some message
> [EE] other message
> [QA] hell if i know what this is
> 
> I've also added more colors to explicitly distinguish einfo from elog,
> and ewarn from eqawarn.  Then, I've replaced most of '>>>' and '!!!'
> used by Portage with four-character versions to keep messages aligned. 
> The 'zings' used for merging files remain three-character, so now it's
> easily possible to distinguish messages from installed file list.
> 
> The PR doing this is: https://github.com/gentoo/portage/pull/759
> 
> Example screenshot:
> https://user-images.githubusercontent.com/110765/135119090-16e9599d-1b0f-41b8-a965-a55577183ffd.png
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] In what phase are file "merged"?

2021-06-23 Thread Fabian Groffen
On 23-06-2021 08:47:58 +0200, Ulrich Mueller wrote:
> >>>>> On Wed, 23 Jun 2021, Joakim Tjernlund wrote:
> 
> > In PMS 9.2 is says:
> > The call order for upgrading, downgrading or reinstalling a package is:
> 
> > pkg_pretend (only for EAPIs listed in table 9.2), which is called 
> > outside of the normal call order process.
> > pkg_setup
> > src_unpack
> > src_prepare (only for EAPIs listed in table 9.3)
> > src_configure (only for EAPIs listed in table 9.4)
> > src_compile
> > src_test (except if RESTRICT=test)
> > src_install
> > pkg_preinst
> > pkg_prerm for the package being replaced
> > pkg_postrm for the package being replaced
> > pkg_postinst
> 
> > It does not say where in this list new files merged and old files
> > unmerged, can anyone enlighten me?
> 
> It's somewhat hidden, but it's there:
> https://projects.gentoo.org/pms/8/pms.html#x1-950009.1.10
> 
>9.1.10 pkg_preinst
>... immediately before merging the package to the live filesystem. ...
> 
>9.1.11 pkg_postinst
>... immediately after merging the package to the live filesystem. ...

Aha, so does this mean pkg_prerm and pkg_postrm are run with replacing
package in place, e.g. if they refer to scripts installed by the
replaced package they may no longer exist or be the same?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] KeyError: 'ED' ?

2021-06-10 Thread Fabian Groffen
On 09-06-2021 14:19:38 -0400, Mike Gilbert wrote:
> On Thu, May 6, 2021 at 6:54 AM Joakim Tjernlund
>  wrote:
> >
> > On Wed, 2021-05-05 at 16:47 +, Joakim Tjernlund wrote:
> > > I am upgrading an old portage(2.3.76) to 3.0.18 using qmerge(binary pkg) 
> > > and I get this:
> > >  qmerge -OK sys-apps/portage
> > > [R] sys-apps/portage-3.0.18
> > >  * Checking for suitable kernel configuration options...
> > >  [ ok ]
> > >  * Using python3.8 in global scope
> > > FEATURES variable contains unknown value(s): disabled
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
> > > return _run_code(code, main_globals, None,
> > >   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
> > > exec(code, run_globals)
> > >   File 
> > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py",
> > >  line 93, in 
> > > main()
> > >   File 
> > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py",
> > >  line 63, in main
> > > config_path = os.path.join(os.environ['ED'], 
> > > GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> > >   File "/usr/lib/python3.8/os.py", line 675, in __getitem__
> > > raise KeyError(key) from None
> > > KeyError: 'ED'
> > >
> > > so portage fails to install properly, I am guessin this is becuse I have 
> > > an old portage to begin with?
> > > Can it be fixed?
> >
> > The error goes away when I do: ED=/ qmerge -OK sys-apps/portage
> > Does that mean that it is qmerge(aka portage-utils) that needs to set ED 
> > during merge? which PHASES ?
> 
> According to PMS, a package manager must define ED in src_install,
> pkg_preinst, and pkg_postinst.

Perhaps qmerge should use export for these vars.  In any case this seems
tracked in bug 792273.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] last-rite: app-admin/diamond

2021-05-24 Thread Fabian Groffen
# Fabian Groffen  (2021-05-24)
# Hack on hack no longer sustainable
# - seemingly dead upstream
# - no (official) Python 3 support
# removal in 30 days, bug: https://bugs.gentoo.org/791613

migrate to something like app-metrics/collectd using the the
write_graphite plugin.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-06 Thread Fabian Groffen
On 06-05-2021 15:01:33 +0200, Andreas K. Huettel wrote:
> > Unfortunately there is not much documentation on "tainted" data for
> > Exim[1], and to resolve this, non-official sources need to be used,
> > such as [2] and [3].
> 
> This is a safety mechanism that is part of Perl (essentially a way of 
> tracking data that is derived from "insecure" sources).
> 
> So it probably would make sense to at least point towards that concept 
> in Perl.

I think the concept is clear to most from the descriptions one can find.
The big problem however is the solution, how to fix one's configuration.

Luckily it seems people find their way to Exim's bugtracker to get help
there.

Thanks for the suggestion though,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [News item review v2] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
Title: Exim>=4.94 transports: tainted not permitted
Author: Fabian Groffen 
Posted: 2021-05-??
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: mail-mta/exim

The Message Transfer Agent Exim disallows tainted variables in transport
configurations since version 4.94.  Existing exim.conf configurations
in /etc/exim need to be reviewed for breakage prior to upgrading to
 >=mail-mta/exim-4.94 to avoid error conditions at runtime.

Since the release of Exim-4.94, transports refuse to use tainted data in
constructing a delivery location.  If you use this in your transports,
your configuration will break, causing errors and possible downtime.

Particularly, the use of $local_part in any transport, should likely be
updated with $local_part_data.  Check your local_delivery transport,
which historically used $local_part.

Unfortunately there is not much documentation on "tainted" data for
Exim[1], and to resolve this, non-official sources need to be used, such
as [2] and [3].



[1] https://lists.exim.org/lurker/message/20201109.222746.24ea3904.en.html
[2] https://mox.sh/sysadmin/tainted-filename-errors-in-exim-4.94/
[3] 
https://jimbobmcgee.wordpress.com/2020/07/29/de-tainting-exim-configuration-variables/

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
On 02-05-2021 12:23:30 +0200, Ulrich Mueller wrote:
> >>>>> On Sun, 02 May 2021, Fabian Groffen wrote:
> 
> > Title: Exim >=4.94 disallows tainted variables in transport configurations
> 
> Title is too long (GLEP 42 allows 50 chars max).

ah, missed that

> I have no idea what this news item is trying to tell me. But I don't use
> Exim, so probably that's the reason. :) Maybe mention at least that Exim
> is a mailer?

Fair point.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
Title: Exim >=4.94 disallows tainted variables in transport configurations
Author: Fabian Groffen 
Posted: 2021-05-??
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: mail-mta/exim

Since the release of Exim-4.94, transports refuse to use tainted data in
constructing a delivery location.  If you use this in your transports,
your configuration will break, causing errors and possible downtime.

Particularly, the use of $local_part in any transport, should likely be
updated with $local_part_data.  Check your local_delivery transport,
which historically used $local_part.

Unfortunately there is not much documentation on "tainted" data for
Exim[1], and to resolve this, non-official sources need to be used, such
as [2] and [3].



[1] https://lists.exim.org/lurker/message/20201109.222746.24ea3904.en.html
[2] https://mox.sh/sysadmin/tainted-filename-errors-in-exim-4.94/
[3] 
https://jimbobmcgee.wordpress.com/2020/07/29/de-tainting-exim-configuration-variables/

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Packages up for grabs: x11-misc/xbindkeys

2021-02-21 Thread Fabian Groffen
Jeroen was sadly removed from the project.

You go ahead and take it.

Fabian

On 22-02-2021 07:49:55 +0200, Jaco Kroon wrote:
> Hi,
> 
> Looks like Jeroen was the person mostly committing on this, Jeroen - are
> you intending to keep maintaining this?
> 
> The PR is legit but incomplete (it merely adds the patch, not use it).
> 
> The patch has been merged upstream in 2014 already, instead we should
> stable 1.8.7.
> 
> I'll adjust the referenced bug referenced by the PR accordingly.
> 
> Kind Regards,
> Jaco
> 
> On 2021/02/21 05:08, Jonas Stein wrote:
> > Dear all
> >
> > the following packages are up for grabs after dropping
> > desktop-misc:
> >
> > x11-misc/xbindkeys
> > https://packages.gentoo.org/packages/x11-misc/xbindkeys
> >
> > There is an open PR
> > https://github.com/gentoo/gentoo/pull/14706
> >
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [PATCH] eclass/db.eclass: use get_libname for Darwin support

2021-01-12 Thread Fabian Groffen
Signed-off-by: Fabian Groffen 
---
 eclass/db.eclass | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/eclass/db.eclass b/eclass/db.eclass
index 9a246d18979..52afe0b765f 100644
--- a/eclass/db.eclass
+++ b/eclass/db.eclass
@@ -23,13 +23,13 @@ db_fix_so() {
cd "${LIB}" || die
 
# first clean up old symlinks
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so' -delete || die
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so.[23]' -delete 
|| die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname)" 
-delete || die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname 
"[23]")" -delete || die
find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*a' -delete || die
 
# now rebuild all the correct ones
local ext
-   for ext in so a; do
+   for ext in so dylib a; do
for name in libdb libdb_{cxx,tcl,java,sql,stl}; do
target="$(find . -maxdepth 1 -type f -name 
"${name}-*.${ext}" |sort -V |tail -n 1)"
[[ -n "${target}" ]] && ln -sf ${target//.\//} 
${name}.${ext}
@@ -37,17 +37,17 @@ db_fix_so() {
done;
 
# db[23] gets some extra-special stuff
-   if [[ -f libdb1.so.2 ]]; then
-   ln -sf libdb1.so.2 libdb.so.2
-   ln -sf libdb1.so.2 libdb1.so
-   ln -sf libdb1.so.2 libdb-1.so
+   if [[ -f libdb1$(get_libname 2) ]]; then
+   ln -sf libdb1$(get_libname 2) libdb$(get_libname 2)
+   ln -sf libdb1$(get_libname 2) libdb1$(get_libname)
+   ln -sf libdb1$(get_libname 2) libdb-1$(get_libname)
fi
# what do we do if we ever get 3.3 ?
local i
for i in libdb libdb_{cxx,tcl,java,sql,stl}; do
-   if [[ -f ${i}-3.2.so ]]; then
-   ln -sf ${i}-3.2.so ${i}-3.so
-   ln -sf ${i}-3.2.so ${i}.so.3
+   if [[ -f $i-3.2$(get_libname) ]]; then
+   ln -sf $i-3.2$(get_libname) $i-3$(get_libname)
+   ln -sf $i-3.2$(get_libname) $i$(get_libname 3)
fi
done
 
@@ -143,8 +143,8 @@ db_src_install_usrlibcleanup() {
mv "${LIB}/libdb_cxx.a" "${LIB}/libdb_cxx-${SLOT}.a" || die
fi
 
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so' -delete || die
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so.[23]' -delete 
|| die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname)" 
-delete || die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname 
"[23]")" -delete || die
einfo "removing unversioned static archives"
find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*a' -delete || die
 
-- 
2.26.2




Re: [gentoo-dev] [RFC] Dealing with global /usr/bin/libtool use vs CC/CXX etc.

2021-01-10 Thread Fabian Groffen
On 10-01-2021 14:34:13 +0100, Michał Górny wrote:
> The vast majority of libtool-based programs use configure script to
> generate libtool.  However, a few non-autoconf packages also use libtool
> by calling system-installed /usr/bin/libtool.  The problem is that this
> libtool hardcodes the values of CC/CXX at its' build time, so unless it
> is rebuilt frequently, packages end up using the stale values.
> The problem is known since 2005 [1] and hasn't been resolved yet.
> 
> I can think of two ways of solving it:
> 
> 1. We could patch system-installed libtool to respect environment
> variables such as CC, CXX, etc.  This will probably require carrying
> a (possibly non-trivial) patch forever.  On the bright side, libtool is
> not exactly a package seeing frequent releases.  I mean, the current
> version is from 2015.
> 
> 2. We could regenerate libtool and force local instance of libtool
> in the packages needing it.  The main advantage of this is that it's
> a no-brainer.  I could make a quick eclass that does configure a local
> instance and prepends it into PATH.
> 
> WDYT?

I would prefer option 2, also because on some systems usr/bin/libtool is
some entirely different tool than GNU libtool.

I remember this being much more of a problem ~15 years ago, so I wonder
do we have an easy way of crafting a list of affected packages, such
that we can see how big the problem actually is nowadays?  I'm thinking
perhaps tinderbox logs or something can reveal /usr/bin/libtool usage
somehow.

Thanks,
Fabian

> [1] https://bugs.gentoo.org/88596


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] acct-user.eclass: don't modify existing user by default

2021-01-08 Thread Fabian Groffen
On 04-01-2021 17:14:47 +0100, Michał Górny wrote:
> Yes, I don't mind an option, as long as it spews a big fat ewarn that
> the user loses the right to support.  However, that's still not
> the right solution to the immediate problem, and I'm currently working
> on a better patch, so I'd prefer if you waited with that to avoid merge
> conflicts.

Could you please share your intended approach?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)

2020-12-05 Thread Fabian Groffen
Thanks Zac!

On 04-12-2020 14:58:22 -0800, Zac Medico wrote:
> Ensure that _get_lock_fn arguments to multiprocessing.Process will
> successfully pickle, as required by the spawn start method, which
> is the default for macOS since Python 3.8.
> 
> Since file descriptors are not inherited unless the fork start
> method is used, the subprocess should only try to close an
> inherited file descriptor for the fork start method.
> 
> Bug: https://bugs.gentoo.org/758230
> Signed-off-by: Zac Medico 
> ---
>  lib/portage/locks.py | 36 +++-
>  1 file changed, 23 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/portage/locks.py b/lib/portage/locks.py
> index 1073343be..193045c03 100644
> --- a/lib/portage/locks.py
> +++ b/lib/portage/locks.py
> @@ -44,18 +44,7 @@ def _get_lock_fn():
>   if _lock_fn is not None:
>   return _lock_fn
>  
> - def _test_lock(fd, lock_path):
> - os.close(fd)
> - try:
> - with open(lock_path, 'a') as f:
> - fcntl.lockf(f.fileno(), 
> fcntl.LOCK_EX|fcntl.LOCK_NB)
> - except EnvironmentError as e:
> - if e.errno == errno.EAGAIN:
> - # Parent process holds lock, as expected.
> - sys.exit(0)
>  
> - # Something went wrong.
> - sys.exit(1)
>  
>   fd, lock_path = tempfile.mkstemp()
>   try:
> @@ -64,8 +53,16 @@ def _get_lock_fn():
>   except EnvironmentError:
>   pass
>   else:
> - proc = multiprocessing.Process(target=_test_lock,
> - args=(fd, lock_path))
> + proc = multiprocessing.Process(
> + target=_subprocess_test_lock,
> + args=(
> + # Since file descriptors are not 
> inherited unless the fork start
> + # method is used, the subprocess should 
> only try to close an
> + # inherited file descriptor for the 
> fork start method.
> + fd if 
> multiprocessing.get_start_method() == "fork" else None,
> + lock_path,
> + ),
> + )
>   proc.start()
>   proc.join()
>   if proc.exitcode == os.EX_OK:
> @@ -80,6 +77,19 @@ def _get_lock_fn():
>   _lock_fn = fcntl.flock
>   return _lock_fn
>  
> +def _subprocess_test_lock(fd, lock_path):
> + if fd is not None:
> + os.close(fd)
> + try:
> + with open(lock_path, 'a') as f:
> + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
> + except EnvironmentError as e:
> + if e.errno == errno.EAGAIN:
> +     # Parent process holds lock, as expected.
> + sys.exit(0)
> +
> + # Something went wrong.
> + sys.exit(1)
>  
>  _open_fds = {}
>  _open_inodes = {}
> -- 
> 2.26.2
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last-rites: sys-kernel/xnu-headers, sys-libs/darwin-libc-headers, dev-libs/libmissing

2020-11-23 Thread Fabian Groffen
# Fabian Groffen  (2020-11-23)
# No longer used, not really functional either, noone should be using
# this, removal in 30 days.
sys-kernel/xnu-headers
sys-libs/darwin-libc-headers
dev-libs/libmissing

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:21:53 +, Alexey Sokolov wrote:
> > What instructed you to perform the migration?  Was it the news-item?  I
> > don't think it should apply for Prefix profiles, and perhaps we should
> > be happy the tool won't work.
> 
> It was the big scary warning about the deprecation whenever I run
> emerge. It contains list of steps.

Ok.  I don't know if we can do anything about that.

> > Did it do anything to your system like creating a lib64 directory?  Does
> > anything work (because I have doubts on whether your system can still
> > find the libs in there now).
> 
> Yes. Attaching logs.

The logs seem to indicate that it thinks all libs on your systems do not
belong to any package.  This suggests the tool cannot locate the VDB or
something, as most of the things in the list are obviously owned by
packages.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:49:27 +0100, Michał Górny wrote:
> > > So what you're saying is that you've had the wrong value of SYMLINK_LIB
> > > for ages, and now you've created a meaningless 17.1 profile that chnages
> > > it but isn't actually supposed to change anything, correct?
> > 
> > I guess, because the amd64 17.0 profile is deprecated with force, and I
> > had to do something ...
> 
> Now that's a lie.  Only the regular amd64 profiles are deprecated.
> There are no deprecation notices e.g. in the x32 profile or prefix
> profiles.

Our profiles either directly depend on the amd64/17.0 profile, or we use
a sub-profile from amd64/17.0 profile, so if it's going to get removed,
we are having a problem, don't we?


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:34:52 +0100, Michał Górny wrote:
> On Tue, 2020-11-10 at 08:55 +0100, Fabian Groffen wrote:
> > On 09-11-2020 19:38:28 +, Alexey Sokolov wrote:
> > > Hi Fabian
> > > I tried to migrate my prefix to 17.1, and there are issues.
> > > 
> > > 1) unsymlink-lib requires "--root ~/gentoo" and otherwise produces an
> > > error "/usr/lib is a real directory! was the migration done already?"
> > 
> > I think unsymlink-lib doesn't have Prefix support, but in addition,
> > what unsymlink-lib is trying to achieve, is not a thing perhaps on
> > Prefix.
> > 
> > A prefix system (at least all of mine) doesn't have libXX or lib/XX
> > (a.k.a.  multilib) directories.  The /usr-split was long ago removed,
> > and thus what we have is:
> > 
> >   lib -> usr/lib
> > 
> > Now, SYMLINK_LIB=no seems to split into lib and lib64, but lib64 does
> > not exist on Prefix systems.
> > 
> 
> So what you're saying is that you've had the wrong value of SYMLINK_LIB
> for ages, and now you've created a meaningless 17.1 profile that chnages
> it but isn't actually supposed to change anything, correct?

I guess, because the amd64 17.0 profile is deprecated with force, and I
had to do something ...


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-09 Thread Fabian Groffen
On 09-11-2020 19:38:28 +, Alexey Sokolov wrote:
> Hi Fabian
> I tried to migrate my prefix to 17.1, and there are issues.
> 
> 1) unsymlink-lib requires "--root ~/gentoo" and otherwise produces an
> error "/usr/lib is a real directory! was the migration done already?"

I think unsymlink-lib doesn't have Prefix support, but in addition,
what unsymlink-lib is trying to achieve, is not a thing perhaps on
Prefix.

A prefix system (at least all of mine) doesn't have libXX or lib/XX
(a.k.a.  multilib) directories.  The /usr-split was long ago removed,
and thus what we have is:

  lib -> usr/lib

Now, SYMLINK_LIB=no seems to split into lib and lib64, but lib64 does
not exist on Prefix systems.

Since Prefix is non-multilib by design*, I wonder if unsymlink-lib is
necessary in the best case, but going to break the Prefix system in the
worst case.

What instructed you to perform the migration?  Was it the news-item?  I
don't think it should apply for Prefix profiles, and perhaps we should
be happy the tool won't work.

* non-multilib is a decision dating back a decade or so, which means
  effectively any Prefix install you encounter should be non-multilib


> 2) $ unsymlink-lib --root ~/gentoo --migrate --pretend
> usage: unsymlink-lib [-h] [-p] [--root ROOT] [--analyze] [--migrate]
>  [--rollback] [--finish] [--force-rollback]
>  [--resume-finish] [-P PREFIX] [--hardlink]
> unsymlink-lib: error: Requested action requires root privileges
> 
> Well, I worked it around by adding "is_root = True" to unsymlink-lib

Did it do anything to your system like creating a lib64 directory?  Does
anything work (because I have doubts on whether your system can still
find the libs in there now).

> 
> 3) Step 9 (Rebuild gcc) fails:
> configure:4372: checking whether the C compiler works
> 
> 
> 
> configure:4394: x86_64-pc-linux-gnu-gccconftest.c  >&5
> 
> 
> 
> /home/user/gentoo/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/as:
> error while loading shared libraries:
> libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so: cannot open shared

Something like this I was suspecting.  Can you still rollback?  If you
can, I'd try that and hope it restores your system in working order.

For any Prefix user, DO NOT run unsymlink-lib, I believe you should NOT
NEED IT!

Thanks,
Fabian

> object file: No such file or directory
> configure:4398: $? = 1
> 
> 
> 
> configure:4436: result: no
> 
> The file exists:
> $ find ~ -name libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so
> /home/user/gentoo/usr/lib/binutils/x86_64-pc-linux-gnu/2.34/libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so
> 
> -- 
> Best regards,
> Alexey "DarthGandalf" Sokolov
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-07 Thread Fabian Groffen
On 07-11-2020 11:42:44 +, Alexey Sokolov wrote:
> 22.10.2020 20:16, Andreas K. Hüttel пишет:
> > Am Donnerstag, 22. Oktober 2020, 18:44:48 EEST schrieb Michał Górny:
> >> On Thu, 2020-10-22 at 11:17 -0400, Brian Evans wrote:
> >>> Users frequently are choosing the wrong profile versions in new installs
> >>> and accidentally downgrading to 17.0 with some saying they see it first.
> >>>
> >>> A simple reordering could help new installs.
> > 
> > Independent of this useful change, it's probably time to deprecate the 
> > amd64 
> > 17.0 profiles!
> > 
> 
> Prefix bootstrap script still makes new installations to use it

This should be solved with

b0445c0a8dd6d2f792c5bb088b154aca53868353
a9c478dc881ee18fefc7342da994b00e60eaad8e

on gentoo.git and

0d7f6b6eb00d0f51f35019846b8f79048b30be93

on prefix.git.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] net-misc/asterisk fails to compile: clang/LLVM: bug 731280

2020-08-28 Thread Fabian Groffen
On 28-08-2020 08:52:09 +0100, Sergei Trofimovich wrote:
> On Fri, 28 Aug 2020 07:37:54 +0100
> Sergei Trofimovich  wrote:
> 
> > On Fri, 28 Aug 2020 08:15:47 +0200
> > Jaco Kroon  wrote:
> > 
> > > Hi All,
> > > 
> > > https://bugs.gentoo.org/731280
> > > 
> > > Summary:
> > > 
> > > This machine uses a clang/LLVM toolchain.
> > > Asterisk fails to compile, ./configure fails with:
> > > 
> > > checking for RAII support... checking for clang -fblocks...
> > > configure: error: BlocksRuntime is required for clang, please install
> > > libblocksruntime
> > > 
> > > I suspect this explains the issue:
> > > 
> > > https://github.com/mackyle/blocksruntime
> > > 
> > > I have no idea how to actually solve this.  
> > 
> > honggfuzz also needs it as it happens to use -fblocks on clang:
> > https://bugs.gentoo.org/729256
> > 
> > We need to package the library. I can give it a try.
> 
> Packaged library as sys-libs/blocksruntime:
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ae376c73ef197d6c7aa619e821c436ccab0cd77e
> 
> Usage example for app-forensics/honggfuzz:
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd841336dfdefbc14907e2d9b1eb1a1a3f5f8b8e

This is cool, but shouldn't it be something like openmp?  (e.g. blocks)

There is no reason blocks have to be used if not on macOS (where system
headers use blocks features).

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] euses(1) Reimplementation

2020-07-09 Thread Fabian Groffen
706#c4
> [4] https://dl.acm.org/doi/abs/10.1145/116825.116845
> [5] 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strcasestr.c;h=d2964c5548b9ea7a68fc5b18b25ddfe7ddd6835c;hb=HEAD#l45
> [6] http://git.suugaku.co.uk/ash-euses/tree/
> [7] http://git.suugaku.co.uk/ash-euses/snapshot/ash-euses-0.3.tar.gz
> 
> -- 
> 
> Ashley Dixon
> suugaku.co.uk
> 
> 2A9A 4117
> DA96 D18A
> 8A7B B0D2
> A30E BF25
> F290 A8AA
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-07-01 Thread Fabian Groffen
On 30-06-2020 13:13:29 -0500, Sid Spry wrote:
> On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote:
> > Hi,
> > 
> > On 29-06-2020 21:13:43 -0500, Sid Spry wrote:
> > > Hello,
> > > 
> > > I have some runnable pseudocode outlining a faster tree verification 
> > > algorithm.
> > > Before I create patches I'd like to see if there is any guidance on 
> > > making the
> > > changes as unobtrusive as possible. If the radical change in algorithm is
> > > acceptable I can work on adding the changes.
> > > 
> > > Instead of composing any kind of structured data out of the portage tree 
> > > my
> > > algorithm just lists all files and then optionally batches them out to 
> > > threads.
> > > There is a noticeable speedup by eliding the tree traversal operations 
> > > which
> > > can be seen when running the algorithm with a single thread and comparing 
> > > it to
> > > the current algorithm in gemato (which should still be discussed here?).
> > 
> > I remember something that gemato used to use multiple threads, but
> > because it totally saturated disk-IO, it was brought back to a single
> > thread.  People were complaining about unusable systems.
> > 
> 
> I think this is an argument for cgroups limits support on the portage process 
> or
> account as opposed to an argument against picking a better algorithm. That is
> something I have been working towards, but I am only one man.

But this requires a) cgroups support, and b) the privileges to use it.
Shouldn't be a problem in the normal case, but just saying.

> > In any case, can you share your performance results?  What speedup did
> > you see, on warm and hot FS caches?  Which type of disk do you use?
> > 
> 
> I ran all tests multiple times to make them warm off of a Samsung SSD, but
> nothing very precise yet.
> 
> % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo
> [...]
> INFO:root:Verifying /var/db/repos/gentoo...
> INFO:root:/var/db/repos/gentoo verified in 16.45 seconds
> 
> sometimes going higher, closer to 18s, vs.
> 
> % ./veriftree.py
> 4.763171965983929
> 
> So roughly an order of magnitude speedup without batching to threads.

That is kind of a change.  Makes one wonder if you really did the same
work.

> > You could compare against qmanifest, which uses OpenMP-based
> > paralllelism while verifying the tree.  On SSDs this does help.
> > 
> 
> I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG
> directory? My code is partially structured as it is because I had problems 
> doing
> this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest.

qmanifest doesn't do much magic out of the standard gnupg practices.
(It is using gpgme.)  If you want it to use a different gnupg dir, you
may change HOME, or GNUPGHOME.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-06-30 Thread Fabian Groffen
: str) -> bool:
> response = dns.resolver.query(domain, dns.rdatatype.NS)
> nsname = response.rrset[0]
> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
> nsaddr = response.rrset[0].to_text()
> 
> # DNSKEY
> request = dns.message.make_query(domain,
> dns.rdatatype.DNSKEY, want_dnssec=True)
> response = dns.query.udp(request, nsaddr)
> if response.rcode() != 0:
> raise Exception('Unable to request dnskey.')
> 
> answer = response.answer
> if len(answer) != 2:
> raise Exception('Malformed answer to dnskey query.')
> 
> name = dns.name.from_text(domain)
> try:
> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
> except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
> #raise exc
> return False
> except AttributeError as exc:
> # Validation may have failed; DNSKEY missing signer attribute. dig 
> may report
> # domain as valid.
> #
> # TODO: Additional state where subdomain of valid domain may fail 
> with 3 nested
> # KeyErrors. Avoid temptation to wildcard catch. Safer to put in 
> process?
> #raise exc
> return False
> else:
> return True
> 
> def hash_localpart(incoming: bytes) -> str:
> '''Z-base32 the localpart of an e-mail address
> 
> 
> https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
> describes why this is needed.
> 
> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
> description of the z-base32 scheme.
> '''
> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
> 
> b = hashlib.sha1(incoming).digest()
> ret = ""
> assert(len(b) * 8 == 160)
> for i in range(0, 160, 5):
> byte = i // 8
> offset = i - byte * 8
> # offset | bits remaining in k+1 | right-shift k+1
> # 3 | 0 | x
> # 4 | 1 | 7
> # 5 | 2 | 6
> # 6 | 3 | 5
> # 7 | 4 | 4
> if offset < 4:
> n = (b[byte] >> (3 - offset))
> else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
> 
> ret += zb32[n & 0b1]
> return ret
> 
> def build_web_key_uri(address: str) -> str:
> local, remote = address.split('@')
> local = hash_localpart(local.encode('utf-8'))
> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
> local
> 
> def stream_to_file(uri: str, fname: str) -> None:
> with requests.get(uri, verify=True, stream=True) as r:
> from pprint import pprint
> pprint(r.headers)
> with open(fname, 'wb') as f:
> shutil.copyfileobj(r.raw, f)
> ```
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Fabian Groffen
Hi Chun-Yu,

On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote:
> Hi,
> 
> I was recently interested in whether portage could be speed up, since
> dependency resolution can sometimes take a while on slower machines.
> After generating some flame graphs with cProfile and vmprof, I found 3
> functions which seem to be called extremely frequently with the same
> arguments: catpkgsplit, use_reduce, and match_from_list.  In the first
> two cases, it was simple to cache the results in dicts, while
> match_from_list was a bit trickier, since it seems to be a requirement
> that it return actual entries from the input "candidate_list".  I also
> ran into some test failures if I did the caching after the
> mydep.unevaluated_atom.use and mydep.repo checks towards the end of the
> function, so the caching is only done up to just before that point.
> 
> The catpkgsplit change seems to definitely be safe, and I'm pretty sure
> the use_reduce one is too, since anything that could possibly change the
> result is hashed.  I'm a bit less certain about the match_from_list one,
> although all tests are passing.
> 
> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world"
> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup.  "emerge
> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec
> (2.5% improvement).  Since the upgrade case is far more common, this
> would really help in daily use, and it shaves about 30 seconds off
> the time you have to wait to get to the [Yes/No] prompt (from ~90s to
> 60s) on my old Sandy Bridge laptop when performing normal upgrades.
> 
> Hopefully, at least some of these patches can be incorporated, and please
> let me know if any changes are necessary.

This sounds like a good job to me!  Do you have any idea what the added
memory pressure for these changes are?

Thanks,
Fabian
> 
> Thanks,
> Chun-Yu
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Use env to find python

2020-06-17 Thread Fabian Groffen
On 16-06-2020 23:10:28 -0500, Sid Spry wrote:
> On Tue, Jun 16, 2020, at 3:57 PM, Michał Górny wrote:
> > '/usr/bin/env python' (with no extra options) is the portable shebang.
> > 
> 
> I added `-S` to preserve the options passed via the shebang line. It seems 
> they can be left off, does anyone know otherwise?

-S is not supported on some platforms.

In any case, I think this patch is wrong.  In the Prefix case, the
shebangs are set during install, in the non-Prefix case I think there is
machinery in place to replace the shebangs during install.

It seems the problem is already solved, why do you need these shebangs
changed?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New USE=-native-symlinks for gcc-config and binutils-config

2020-05-24 Thread Fabian Groffen
On 24-05-2020 10:48:47 +0100, Sergei Trofimovich wrote:
> On Sat, 23 May 2020 22:41:02 -0400
> Mike Gilbert  wrote:
> > I don't think we
> > want to give people the impression that this is a well-supported
> > configuration. I would only expect people to disable this if they want
> > to break their system intentionally.
> 
> Yeah, today it's certainly a way to get your system in a miserable state.
> My hope is that USE=-native-symlinks can get you a working Gentoo in a
> near future by only fixing real package problems and limitations of their
> build systems.

Portage adds PREROOTPATH, ROOTPATH and a standard set of
/usr/{local/,}{s,}bin to PATH in _doebuild_path.  That means if you add
something like /usr/bin/native-toolchain to PATH only, users will get
gcc, ld, etc., while root and Portage will lack these.  One can wonder
if root should have direct access to compilers, but it's easy enough to
add the compilers to PATH if really necessary.

I guess what I'm trying to say is: you can hide effect of the setup for
users if you'd like.  That is, after we had buildbots point out the bulk
of packages that are wrong of course.

Thanks,
Fabian
 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-23 Thread Fabian Groffen
On 22-05-2020 21:58:54 +0200, Michał Górny wrote:
> Let's put it like this.  This thing starts working.  Package X is
> broken, and we see that almost nobody is using it.  We remove that
> package.  Now somebody is angry.  He submits a lot of fake data to
> render the service useless so that we don't make any future decisions
> based on it.

I'm affraid that has a heroic flair to me.  The service should never be
used for decisions like that, because it's a biased sample at most.
Doing stuff like this simply destroys the soul of the distribution.

I hope this isn't one of your genuine objectives with the service.  If
it is, I can see why you fear spam so much.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-21 Thread Fabian Groffen
 alternative of using a proof-of-work algorithm was suggested to me
> yesterday.  The idea is that every submission has to be accompanied with
> the result of some cumbersome calculation that can't be trivially run
> in parallel or optimized out to dedicated hardware.
> 
> On the plus side, it would rely more on actual physical hardware than IP
> addresses provided by ISPs.  While it would be a waste of CPU time
> and memory, doing it just once a week wouldn't be that much harm.
> 
> On the minus side, it would penalize people with weak hardware.
> 
> For example, 'time hashcash -m -b 28 -r test' gives:
> 
> - 34 s (-s estimated 38 s) on Ryzen 5 3600
> 
> - 3 minutes (estimated 92 s) on some old 32-bit Celeron M
> 
> At the same time, it would still permit a lot of fake submissions.  For
> example, randomx [1] claims to require 2G of memory in fast mode.  This
> would still allow me to use 7 threads.  If we adjusted the algorithm to
> take ~30 seconds, that means 7 submissions every 30 s, i.e. 20k
> submissions a day.
> 
> So in the end, while this is interesting, it doesn't seem like
> a workable anti-spam measure.
> 
> 
> Option 3: explicit CAPTCHA
> ==
> A traditional way of dealing with spam -- require every new system
> identifier to be confirmed by solving a CAPTCHA (or a few identifiers
> for one CAPTCHA).
> 
> The advantage of this method is that it requires a real human work to be
> performed, effectively limiting the ability to submit spam.
> The disadvantage is that it is cumbersome to users, so many of them will
> just resign from participating.
> 
> 
> Other ideas
> ===
> Do you have any other ideas on how we could resolve this?
> 
> 
> [1] https://github.com/tevador/RandomX
> 
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-19 Thread Fabian Groffen
On 18-05-2020 18:42:24 -0700, Alec Warner wrote:
> TL;DR: What if we launched id.gentoo.org[1], an identity provider that 
> provides
> authentication for Gentoo properties? Basically, 1 username / password for 
> wiki,
> bugs, email, forums, and any other http service[0][1].

I'd be in favour of SSO for all http-, imap- and smtp-based Gentoo services.

Thanks,
Fabian

> 
> Today Gentoo has numerous systems that mostly work in a segmented way.
> 
>  - To connect to hosts, we use ssh keys.
>  - Git is authenticated via ssh keys.
>  - Email uses LDAP passwords.
>  - Bugzilla has its own identities, with their own passwords.
>  - Wiki is separate, with its own passwords.
>  - Forums are separate.
>  - Infra has an additional 4 systems that use separate credentials.
> 
> Some applications support 2FA (such as wiki.)
> Some applications do not support 2FA.
> Applications that require 2FA have a configuration for each app, so you have N
> configurations.
> 
> If we configured id.gentoo.org[2] you would have 1 identity across all gentoo
> properties.
> 
> Is this a thing people are interested in?
>  
> [0] It's unlikely operations for git via ssh would change in this rollout.
> [1] Its unclear if the scope is "gentoo developers" or "any community member."
> The former have LDAP accounts and @gentoo.org[3] email addresses and so we can
> manage them easily; managing 1000s of other accounts in the IDP remains to be
> seem.
> 
> 
> References
>1. http://id.gentoo.org
>2. http://id.gentoo.org
>3. http://gentoo.org

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Last standing Python 2.7 dependency

2020-05-03 Thread Fabian Groffen
On 02-05-2020 23:24:42 -0700, Brian Dolbec wrote:
> On Sun, 3 May 2020 07:28:50 +0200
> Viktar Patotski  wrote:
> 
> > Hi all,
> > 
> > I'd also like to clean my system and have it Python 2.7 free. Are
> > there any guidelines to check which packages are still using pyton2_7
> > in my system?
> > 
> > Thanks,
> > Viktar
> > 
> 
> There are both equery and enalyze commands in gentoolkit that can give
> you reports about what pkgs are installed.
> 
> equery hasuse
> enalyze analyze [use|pkguse]
> 
> for help on them:
> equery -h
> equery hasuse -h
> enalyze -h
> enalyze a -h

In addition to these great tools, portage-utils' quse might also be
useful:

% quse python2_7
...


Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

2020-04-26 Thread Fabian Groffen
On 26-04-2020 21:29:42 +0200, Michał Górny wrote:
> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> > Bug: https://bugs.gentoo.org/715108
> > Signed-off-by: Matt Turner 
> > ---
> > Strawman patch. Bikeshed away.
> > 
> 
> xz is generally slow and doesn't do parallel good.  If we want to change
> this, we should go for something cool like zstd that scales better.

I'd go for zstd too.  It seems to be the best of both worlds, good
compression at a good speed.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] fcaps.eclass: disable fcaps() on Prefix.

2020-03-07 Thread Fabian Groffen
On 08-03-2020 15:26:50 +0800, hero...@gentoo.org wrote:
> From: Benda Xu 
> 
> Gentoo Prefix runs with a normal user and cannot grant extra
> capabilities.  Exit gracefully with a message.
> 
> Signed-off-by: Benda Xu 
> ---
>  eclass/fcaps.eclass | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/eclass/fcaps.eclass b/eclass/fcaps.eclass
> index 467f955f5e9a..563d177c92d5 100644
> --- a/eclass/fcaps.eclass
> +++ b/eclass/fcaps.eclass
> @@ -78,6 +78,11 @@ DEPEND="filecaps? ( sys-libs/libcap )"
>  fcaps() {
>   debug-print-function ${FUNCNAME} "$@"
>  
> + if [[ ${EUID} != 0 ]] ; then
> + einfo "Insufficient privileges to execute ${FUNCNAME[0]}"
> + return 0

Perhaps add "ignoring" or something in the message here to make it clear
this isn't treated as an error?

Thanks,
Fabian

> + fi
> +
>   # Process the user options first.
>   local owner='root'
>   local group='0'
> -- 
> 2.25.0
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Keywordreqs and slacking arch teams

2019-12-28 Thread Fabian Groffen
On 28-12-2019 22:27:02 +1300, Kent Fredric wrote:
> On Sat, 28 Dec 2019 08:09:33 +0100
> Michał Górny  wrote:
> 
> > How can we improve this?
> 
> Every time this kind of issue comes up, I can't help feeling we need
> some sort of tool more advanced than what we currently have.
> 
> Something that maintains persistence of keyword demands similar to how
> the current bot does, but more ... 
> 
> Its the sort of thing I get tempted to implement myself, but get too
> horrified about the prospect of working with portage internals to do it.

Hmmm, interested to hear what kind of things you're thinking about here.

I feel it would help if we would have the ability to at least
compile/test ebuilds automatically.  Not sure how helpful qemu could be
there, but I know things like compiling for things like arm (RPi) works
reasonably well.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote:
> On 12/13/19 9:28 AM, Fabian Groffen wrote:
> > 
> > We are providing those patches, maybe.  In reality very often the
> > patches originate from somewhere else though.  And you don't want to
> > have to respin all of those just because.  At least that's what I feel.
> > 
> 
> Just because... the context changed? A new "!" in a line of context can
> be the difference between letting someone log in with the right password
> and letting them log in with the wrong password. You should at least
> have to stop and verify that the patch does what you think it does when
> it "gains" fuzz. And at that point, git diff will give you a clean
> version of it.

Counter argument is that we've been doing this for decades, and always
relied on maintainers to check the contents of their patches, without
problems.  We didn't introduce a predictable random number generator or
something.
Your very specific example just illustrates the niche this proposal is
targetting.

As with many of the proposals lately, they just seem to aim at more work
for individual maintainers, with a very low gain ratio.
Better, allow this to be a FEATURE, or whatever, that devs should enable,
but don't spit this in the user's face by default.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 15:24:40 +0100, Michał Górny wrote:
> > > ...and why do we consider it correct to apply patches when the context
> > > doesn't match?  If our only goal is to make things 'easier' for
> > > 'everyone', then we could just pass -F and ignore all the context.
> > > 
> > > Though I don't understand why include any context in the first place if
> > > you don't care about it matching.  Sounds like a waste of space to me!
> > 
> > The patch command defaults to -F2. If that makes no sense, why is it
> > the upstream default?
> > 
> 
> You should ask upstream, not me.  But if I were to guess, the answer
> would be because patch(1) is used by random people trying to apply
> random patches they've found somewhere.  We on the other hand are
> applying patches that *we* are supposed to provide.

We are providing those patches, maybe.  In reality very often the
patches originate from somewhere else though.  And you don't want to
have to respin all of those just because.  At least that's what I feel.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:45:34 +0100, Michał Górny wrote:
> On Tue, 2019-10-29 at 15:33 +0100, Fabian Groffen wrote:
> > In addition, there are currently files there that aren't referenced from
> > ebuilds.  Prefix uses these files during bootstrap, local mirrors are
> > often much faster than dev.g.o.
> > 
> > If the files don't get mirrored anymore, I guess I can create a dummy
> > ebuild that has the files in SRC_URI.
> 
> Ok, this is something I wasn't aware of.  I agree that dummy ebuild
> should not be necessary here.  However, I'm also not sure if distfiles-
> local is really the proper way either, especially that I don't see such
> files on woodpecker right now.

There should be /space/distfiles-local and
/space/distfiles-whitelist/prefix with a list of files to retain on the
mirror.

Thanks,
Fabian

> I don't think the matter is urgent right now, so let's ponder on it
> a bit.  In particular, I think we should have a clear indication of who
> added which files, when, what for and where they came from.  Those are
> precisely the things that the current distfiles-local approach misses.
> 
> > If the files get mirrored, but put in a subdir based on the filename
> > hash, the original query endpoint on distfiles.g.o changes, much like
> > the SRC_URI approach.
> > 
> > Now I can use distfiles.prefix.b.n which redirects to the distfiles.g.o
> > URL with subdir for most part I think, but it's sub-optimal from my
> > point of view.  Calculating the hash is not always feasible due to the
> > lack of b2sum or other means.  Hence my earlier request to have such
> > official translation service on Gentoo hardware.
> > 
> > (I just wrote a small wsgi script that calculates the hash and generates
> > the redirect from Python, served via uwsgi/nginx, but there should be
> > many ways to achieve the same goals, if and only if a blake2b
> > implementation were available for it.)
> 
> This is also something that needs thinking.  I personally don't mind
> having one but it would be nice if it was able to account for geodns
> and such.
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:17:38 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 29 Oct 2019, Michał Górny wrote:
> 
> > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote:
> >> > What if the file is hosted at a non-standard tcp port upstream
> >> > (like http://example.org:8080/)? The devmanual says that it _must_
> >> > be manually uploaded to /space/distfiles-local/ in such cases.
> 
> >> Or another example, app-emacs/vhdl-mode-3.38.1, where (incompetent,
> >> or nasty?) upstream blocks wget for some reason, but other methods
> >> (e.g., curl, firefox) work? How would I get the file onto the mirrors
> >> there?
> 
> > If I were you, I would've explicitly mirrored the file anyway.
> > If upstream blocks wget, then users who do not use GENTOO_MIRRORS will
> > also suffer due to it.
> 
> All what I'm saying is that there can be unusual circumstances where
> manual uploading of a file is useful. So please don't take that
> possibility away.

In addition, there are currently files there that aren't referenced from
ebuilds.  Prefix uses these files during bootstrap, local mirrors are
often much faster than dev.g.o.

If the files don't get mirrored anymore, I guess I can create a dummy
ebuild that has the files in SRC_URI.

If the files get mirrored, but put in a subdir based on the filename
hash, the original query endpoint on distfiles.g.o changes, much like
the SRC_URI approach.

Now I can use distfiles.prefix.b.n which redirects to the distfiles.g.o
URL with subdir for most part I think, but it's sub-optimal from my
point of view.  Calculating the hash is not always feasible due to the
lack of b2sum or other means.  Hence my earlier request to have such
official translation service on Gentoo hardware.

(I just wrote a small wsgi script that calculates the hash and generates
the redirect from Python, served via uwsgi/nginx, but there should be
many ways to achieve the same goals, if and only if a blake2b
implementation were available for it.)

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 05:27:37 +0100, Michał Górny wrote:
> On Tue, 2019-10-29 at 00:24 +0100, Chí-Thanh Christopher Nguyễn wrote:
> > Hi!
> > 
> > > Today you get chastised for using /space/distfiles-local and not
> > > following policy changes.  The devmanual states that it's deprecated
> > > since at least 2011, and talks of using d.g.o [1].
> >  > [1] 
> > https://devmanual.gentoo.org/general-concepts/mirrors/index.html#suitable-download-hosts
> > 
> > Sorry I'm late to the party, but I would like to enquire about what happens 
> > if a file with existing filename but different b2sum gets uploaded to 
> > /space/distfiles-local now?
> 
> The same as before.  It gets put in top-level disfiles directory. 
> Hashes are calculated from filenames, so this wouldn't affect it.  That
> is, if it put those files in subdirectories in the first place because
> it doesn't.

/space/distfiles-local is no longer copied to the mirrors? or just not
copied in the subdir-hierarchy?

> > Doing so and updating the Manifest used to be another (not necessarily 
> > preferred) method to address upstream remaking release packages.
> > 
> 
> It's no longer valid.

Just wondering.  Do you mean it isn't valid that some upstreams do this
(yes horror)?  We surely need a way to work around that ...

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Fabian Groffen
Hi,

On 18-10-2019 15:41:32 +0200, Michał Górny wrote:
> 3. Directly fetching files from distfiles.gentoo.org will become
> a little harder.  To fetch a distfile named 'foo-1.tar.gz', you'd have
> to use something like:
> 
> $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
> 1b
> $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz
> ...
> 
> 
> Alternatively, you can:
> 
> $ wget http://distfiles.gentoo.org/distfiles/INDEX
> 
> and grep for the right path there.  This INDEX is also a more
> lightweight alternative to HTML indexes generated by the servers.

Would it be possible to run a service that sends a 302 for the
distfiles/foo-1.tar.gz to the appropriate bucket such that manual
fetching doesn't require to calculate the hash?

I prototyped this myself for distfiles.prefix, and seems like a nice
guesture for at least the transition period?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Underscores in USE flags

2019-09-21 Thread Fabian Groffen
On 21-09-2019 09:06:01 +0200, Michał Górny wrote:
> On Sat, 2019-09-21 at 08:43 +0200, Fabian Groffen wrote:
> > Why not teach our tools (equery, quse, etc.) to print these USE-flags
> > like Portage does?  (looking them up to be valid expands)
> > Then users have nothing to be confused about (no distinction between
> > foo_bar and FOO="bar"), and new USE_EXPANDS cannot be
> > silently/accidentially introduced.
> 
> I don't see how that solves the problem.  More tools having distinct
> output don't change the fact that anyone with a bit of ebuild knowledge
> will say 'this looks like USE_EXPAND' while looking at it, independently
> of what some tools would say.

Well... someone with a bit of ebuild knowledge would see odd USE-flags.
USE_EXPAND is a (bad) hack, of having some USE-flags mean something
different, or resolve through something different, while in reality they
really don't do anything odd.  In fact, sometimes users have to use
FOO="bar" (make.conf), while other times foo_bar needs to be used
(e.g. use.mask, or IUSE=).

Consistency would be nice, the real question is, what does USE_EXPAND
actually try to achieve, and can we fix it properly in the next EAPI,
such that repoman can also do the proper complaints about USE-flag
(and USE_EXPAND-flag) naming by then.

Back to the thread, the point is, these flags exist today, and renaming
flags is not something to be considered harmless.  As much as the recent
renaming of lm_sensors to lm-sensors caused breakage (and still does,
apparently some tools keep caches, etc.) also renaming USE-flags goes by
problems, in particular for managed systems (Chef, Puppet).  It's not a
matter of just fixing the name for a USE-flag.  This is saying nothing
about whether or not we'd want to change the flag.  It's about the
impact of the change, and whether that is worth it for the noble aim of
consistency or correctness.  I believe this was the OPs point in this
thread.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Underscores in USE flags

2019-09-21 Thread Fabian Groffen
On 20-09-2019 22:53:53 +0200, Michał Górny wrote:
> On Fri, 2019-09-20 at 13:46 -0700, Zac Medico wrote:
> > 
> > If we take this underscore rule to its logical extreme, then we should
> > rename python_targets_python3_7 to python_targets_python3-7, yes?
> 
> Believe me, I would have done that already if not the fact that with all
> the dependency logic around here it would be totally destructive to all
> Gentoo systems.

Honestly, with this reasoning, why force other packages to go through
USE-flag renaming in that case?  A major consumer of USE_EXPAND isn't
sticking to the rule, which makes any benefit of it moot.  Tools cannot
assume the last underscore separates the USE_EXPAND var from its value,
users cannot see what is the value either, without knowledge.

Why not teach our tools (equery, quse, etc.) to print these USE-flags
like Portage does?  (looking them up to be valid expands)
Then users have nothing to be confused about (no distinction between
foo_bar and FOO="bar"), and new USE_EXPANDS cannot be
silently/accidentially introduced.

> But hey, expect hyphen on 3.8.

I honestly feel for consistency and not confusing users, we should
either do them all or stick to the current scheme.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last rites: app-portage/hashgen

2019-09-15 Thread Fabian Groffen
# Fabian Groffen  (2019-09-15)
# Incorporated in >=app-portage/portage-utils-0.80 as qmanifest
# Removal in 30 days.  Bug #694428
app-portage/hashgen


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] use.desc: Introduce global USE=magic

2019-08-20 Thread Fabian Groffen
Hi Michał,

It looks like you missed two from your original set?

app-editors/nano[magic] Add magic file support (sys-apps/file) to
automatically detect appropriate syntax highlighting
www-servers/pshs[magic] Enable automatic detection of Content-Type using
libmagic (sys-apps/file)

Thanks,
Fabian

On 19-08-2019 22:04:12 +0200, Michał Górny wrote:
> On Sun, 2019-08-11 at 13:21 +0200, Michał Górny wrote:
> > USE=magic is currently used consistently by 12 packages:
> > 
> 
> Merged.
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Adding extra vars to md5-cache, for QA purposes

2019-07-26 Thread Fabian Groffen
Hi,

On 25-07-2019 14:20:50 +0200, Michał Górny wrote:
> Hi,
> 
> TL;DR: I'd like to make it possible for ebuilds to define additional
> variables that will be stored in md5-cache.  This would be useful for CI
> and other tooling that right now has to parse ebuilds for other data.

Only downside I can think of, is a diskspace increase for the md5-cache.
Not sure if this is going to be substantial, but given things like
PYTHON_COMPAT, perhaps a quick calculation of extra "cost" can be made.
Should diskspace become a problem, one could consider to use a separate
file/dir, that users could rsync-exclude, since Portage won't need it to
operate properly.

Thanks,
Fabian

> 
> 
> The idea is to add a new incremental ebuild/eclass variable (technical
> name: QA_EXTRA_CACHE_VARS) that would define additional data to be
> stored in cache.  For example, python*-r1 eclasses would define
> 'PYTHON_COMPAT', acct-user would define 'ACCT_USER_ID', etc.
> 
> When regenerating cache, the PM would read this variable, and store
> the values of all defined variables into md5-cache.  As a result,
> programs needing those variables can get them straight from cache
> without having to attempt to run or parse ebuilds (which is both slow
> and prone to bugs).
> 
> This would benefit e.g. gpyutils that right now need to attempt to parse
> PYTHON_COMPAT from ebuilds.  It would also benefit writing future
> pkgcheck checks for user/group ID collisions.
> 
> 
> Notes:
> 
> - since md5-cache uses key-value format and allows for future
> extensions, the new values can be added without breaking anything;
> 
> - md5-cache is not specified in the PMS, and the whole thing can be
> implemented without need for EAPI bump,
> 
> - I would like to have this implemented consistently both in Portage
> and pkgcore,
> 
> - we will need to clearly define how to dump arrays.
> 
> 
> What do you think?
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] New QA policy: Packages must not disable installing manpages via USE flags

2019-07-21 Thread Fabian Groffen
On 21-07-2019 23:47:02 +1200, Kent Fredric wrote:
> Yes, yes, I'm suggesting something perverted like a build server or
> system for aggregating built man-pages on gentoo-servers automatically
> as part of CI, that end users can just trivially fetch. But that's just
> one approach, surely, there are others.

Right, or we go for some (official) form of binpkgs distribution.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Re: RFC: isodate for packages.mask starting on 2019-07-01

2019-06-29 Thread Fabian Groffen
I think ISO 8601 date format is an improvement.

However, as you're suggesting to use a date at UTC time, this begs for
scripting, such as the old echangelog did.  Then, when scripted, the
actual date format is no longer an issue.  Somehow the current format
looks easier to read (for humans) to me, so no real need to change if
there was a script to just add/remove/update entries.  (And commit
without forgetting to add --signoff.)

Fabian

On 29-06-2019 13:23:10 +0200, Jonas Stein wrote:
> Dear all,
> 
> Situation:
> We have different date formats in packages.mask.
> 
> 
> Change:
> I suggest that we start using the date format -mm-dd
> for all dates in packages.mask
> starting with 2019-07-01
> 
> The following changes in packages.mask will introduce the date format,
> specify the timezone, and use Larry as example user.
> 
> 3,6c3,6
> < # When you add an entry to the top of this file, add your name, the
> date, and
> < # an explanation of why something is getting masked. Please be extremely
> < # careful not to commit atoms that are not valid, as it can cause
> large-scale
> < # breakage, especially if it ends up in the daily snapshot.
> ---
> > # When you add an entry to the top of this file, add your name, the date
> > # in the UTC timezone, and an explanation of why something is getting
> masked.
> > # Please be extremely careful not to commit atoms that are not valid,
> as it can
> > # cause large-scale breakage, especially if it ends up in the daily
> snapshot.
> 10c10
> < ## # Dev E. Loper  (28 Jun 2012)
> ---
> > ## # Larry the cow  (2019-07-01)
> 24,26c24,26
> < ## # Dev E. Loper  (23 May 2015)
> < ## # Masked for removal in 30 days.  Doesn't work
> < ## # with new libfoo. Upstream dead, gtk-1, smells
> ---
> > ## # Larry the cow  (2019-07-01)
> > ## # Masked for removal after 2019-08-01.
> > ## # Doesn't work with new libfoo. Upstream dead, gtk-1, smells
> 
> 
> Reason:
> * Larry is the Gentoo Example
> * 2019-01-01 + 30 days is unclear, if we do not use UTC time
> * The new date format is easy to read and write and easy to parse
> internationally.
> 
> Do you have any objections?
> 
> 
> By the way, you can get a formatted string of now in UTC with:
> date -u +"%Y-%m-%d"
> 
> -- 
> Best,
> Jonas
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecation and removal of 13.0 profiles is imminent

2019-02-20 Thread Fabian Groffen
On 19-02-2019 22:52:10 +, Sergei Trofimovich wrote:
> A few months passed and we almost finished with a long tail.
> 
> To-be-deprecated profiles yet are:
>   https://bugs.gentoo.org/673278
> prefix/linux-standalone/ppc64
> 
> +prefix@, what is our plan here? Can we drop at least empty
> 'deprecated' file to notify users?

I cleaned it up, it isn't in use any more.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-11 Thread Fabian Groffen
Pushed, thanks

On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects.  Adjust this interval to
> > be time based.  This means that modern (fast) machines likely will never
> > see the countdown messages at all.  On slow setups the message will be
> > informative that there is progress, albeit rather slowly.  While at it,
> > report percentage done.
> > 
> > Output before this patch:
> > 
> >  * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> > 
> > Possible output after this patch on a slower machine:
> > 
> >  * checking 6158 files for package collisions
> >  48% done,  3145 files remaining ...
> >  96% done,   192 files remaining ...
> > 100% done
> > 
> > Signed-off-by: Fabian Groffen 
> > ---
> >  lib/portage/dbapi/vartree.py | 15 +--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> > 'portage.util.install_mask:install_mask_dir,InstallMask',
> > 'portage.util.listdir:dircache,listdir',
> > 'portage.util.movefile:movefile',
> > +   'portage.util.monotonic:monotonic',
> > 'portage.util.path:first_existing,iter_parents',
> > 'portage.util.writeable_check:get_ro_checker',
> > 'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > totfiles = len(file_list) + len(symlink_list)
> > +   previous = monotonic()
> > +   progress_shown = False
> > +   report_interval = 1.7  # seconds
> > +   falign = len("%d" % totfiles)
> > showMessage(_(" %s checking %d files for package 
> > collisions\n") % \
> > (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > -   if i % 1000 == 0 and i != 0:
> > -   showMessage(_("%d files remaining 
> > ...\n") % (totfiles - i))
> > +   current = monotonic()
> > +   if current - previous > report_interval:
> > +   showMessage(_("%3d%% done,  %*d files 
> > remaining ...\n") %
> > +   (i * 100 / totfiles, 
> > falign, totfiles - i))
> > +   previous = current
> > +   progress_shown = True
> >  
> > dest_path = normalize_path(
> > os.path.join(destroot, 
> > f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> > break
> > if stopmerge:
> > collisions.append(f)
> > +   if progress_shown:
> > +   showMessage(_("100% done\n"))
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> Looks good!
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   previous = monotonic()
+   progress_shown = False
+   report_interval = 1.7  # seconds
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   current = monotonic()
+   if current - previous > report_interval:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
falign, totfiles - i))
+   previous = current
+   progress_shown = True
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if progress_shown:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > +   if tnow + tinterv < ninterv:
> > +   showMessage(_("100% done\n"))
> 
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
> 
> 
> previous = monotonic()
> progress_shown = False

You're right.  I should've gone for readability in the first place.
There is no point in trying to save anything here.

Thanks,
Fabian

> for f in files:
>   current = monotonic()
>   if current - previous > tinterv:
>   showMessage(...)
>   previous = current
>   progress_shown = True
> 
> if progress_shown:
>   showMessage(_("100% done\n"))
> 
> 
> 
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> 
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = monotonic()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if monotonic() > ninterv:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (falign, i * 100 / 
totfiles, totfiles - i))
+   ninterv = monotonic() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
On 08-01-2019 20:59:34 +, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> >> be time based.  This means that modern (fast) machines likely will never
> >> see the countdown messages at all.  On slow setups the message will be
> >> informative that there is progress, albeit rather slowly.  While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >>  * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >>  * checking 6158 files for package collisions
> >>  48% done, 3145 files remaining ...
> >>  96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen 
> >> ---
> >>  lib/portage/dbapi/vartree.py | 11 +--
> >>  1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>symlink_collisions = []
> >>destroot = self.settings['ROOT']
> >>totfiles = len(file_list) + len(symlink_list)
> >> +  tnow = time.time()
> >> +  tinterv = 2  # seconds
> >> +  ninterv = tnow + tinterv
> >>showMessage(_(" %s checking %d files for package 
> >> collisions\n") % \
> >>(colorize("GOOD", "*"), totfiles))
> >>for i, (f, f_type) in enumerate(chain(
> >>((f, "reg") for f in file_list),
> >>((f, "sym") for f in symlink_list))):
> >> -  if i % 1000 == 0 and i != 0:
> >> -  showMessage(_("%d files remaining 
> >> ...\n") % (totfiles - i))
> >> +  if time.time() > ninterv:
> >> +  showMessage(_("%3d%% done, %d files 
> >> remaining ...\n") %
> >> +  (i * 100 / totfiles, 
> >> totfiles - i))
> >> +  ninterv = time.time() + tinterv
> >>  
> >>dest_path = normalize_path(
> >>os.path.join(destroot, 
> >> f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>break
> >>if stopmerge:
> >>collisions.append(f)
> >> +  if tnow + tinterv < ninterv:
> >> +  showMessage(_("100% done\n"))
> >>return collisions, dirs_ro, symlink_collisions, 
> >> plib_collisions
> >>  
> >>def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().

Will do.

> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
> 
>  * Final size of build directory: 2696 KiB (2.6 MiB)
>  * Final size of installed tree:  5372 KiB (5.2 MiB)
> 
> Otherwise, I think this will be quite helpful. Thanks.

I think I understand what you mean, let me see if I can do something
without too much complexity.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done, 3145 files remaining ...
 96% done, 192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = time.time()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if time.time() > ninterv:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
+   ninterv = time.time() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote:
> 
> > Output before this patch:
> 
> >  * checking 6111 files for package collisions
> >  [...]
> 
> > After:
> 
> >  * checking 6158 files for package collisions
> >  [...]
> 
> Why does the total number of files change?

First was python2, second python3.  The result was indicative only, the
number doesn't change of course.

Sorry for the confusion.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
 16% done, 5131 files remaining ...
 33% done, 4104 files remaining ...
 50% done, 3077 files remaining ...
 66% done, 2050 files remaining ...
 83% done, 1023 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
-   (colorize("GOOD", "*"), totfiles))
+   (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if bucksize < totfiles:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

2019-01-07 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%2d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
-- 
2.20.1




Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format

2018-11-27 Thread Fabian Groffen
On 26-11-2018 21:13:53 +, Andrey Utkin wrote:
> On Wed, Nov 21, 2018 at 11:45:54AM +0100, Fabian Groffen wrote:
> > We agree it is hackish, and we agree we can do without.  You simply
> > exaggerate the problem, IMO, which mostly isn't there, because it works
> > fine today.  It can also be solved today using shell tools.
> 
> I am sad that you don't see it as a productivity impediment that the
> user is required to know the custom tooling to do even such a trivial
> non-standard action as manual extraction.

Huh?  tar -jxf doesn't do the trick for you?

> Maybe I will make myself look bad by admitting this, but I'm not meeting
> your expectations. I use Gentoo for ~11 years, and for about one year I
> am using my private binpkgs distributed to all my machines (i.e. I have
> read binary package guide fair number of times, but I stopped rereading
> it when I satisfied my needs). When in need, I still reached to trusty
> tar, and I did not even know what are the names of special tools (a
> toolchain?) qtbz2 and qxpak.
> 
> Just few days ago I messed with binpkgs for investigation purpose. I
> just wanted to extract few to somewhere (definitely not into system
> root), and read a core dump with GDB asking it to use those extracted
> files for debug symbols.
> 
> Of course I used `tar xaf`, because what I know is that it's honest tbz2
> just with metadata appended.
> 
>  # tar xaf boost-1.65.0.tbz2
> 
> bzip2: (stdin): trailing garbage after EOF ignored
> 
> Exit code is 0.
> But the notice is annoying (on subconscious level), because Silence Is
> Golden - "when a program has nothing interesting or surprising to say,
> it should shut up".

You seem to contradict yourself.  You didn't know the tools, yet you say
you needed to, to unpack the files.  But you show here you just unpacked
the files without said knowledge.

> > % head -c `grep -abo 'XPAKPACK' 
> > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | sed 's/:.*$//'` 
> > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | tar -jxf -
> > 
> > results in no warnings/errors from bzip about trailing garbage, possible
> > thanks to the spec being smart enough about this.
> 
> Thanks, this is a very concise **custom tool** to handle current binpkg
> format.

As is tar followed by tar.  The obvious advantage of the latter is that
you don't get a warning which could trigger you into thinking something
is wrong.  So, in my opinion, that is a better way of doing it compared
to the current way.

> > Not having to do this, when under stress and pressure to restore a
> > system to get it back into production, is a plus.  Though, in that
> > scenario the trailing garbage warning wouldn't have been that bad
> > either.
> 
> When understress and pressure, the irrelevant warning is not bad?
> I am sure it is really bad for operator's attention.

I've been using Gentoo binpkgs for a long while, I think something like
~14 years ago when I used them extensively.  Perhaps I'm an exception,
but back then I knew already there was an extra bit attached to the
tars, as were all my collegues around me back then.  The fact it comes
up now (as a surprise?) maybe means the knowledge has gone.  So good
thing we're replacing it with something easier to infer from inspecting
it.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [pre-GLEP r2] Gentoo binary package container format

2018-11-21 Thread Fabian Groffen
On 20-11-2018 21:33:17 +0100, Michał Górny wrote:
> The volume label
> 
> 
> The volume label provides an easy way for users to identify the binary
> package without dedicated tooling or specific format knowledge.
> 
> The implementations should include a volume label consisting of fixed
> string ``gpkg:``, followed by a single space, followed by full package
> identifier.  However, the implementations must not rely on the volume
> label being present or attempt to parse its value when it is.
> 
> Furthermore, since the volume label is included in the .tar archive
> as the first member, it provides a magic string at a fixed location
> that can be used by tools such as file(1) to easily distinguish Gentoo
> binary packages from regular .tar archives.

Just for clarity on this point.
Are you proposing that we patch file(1) to print the Volume Header here?
file-5.35 seems to not say much but "tar archive" or "POSIX tar archive"
for tar-files containing a Volume Header as shown by tar -tv.

> Container and archive formats
> -
> 
> During the debate, the actual archive formats to use were considered.
> The .tar format seemed an obvious choice for the image archive since
> it is the only widely deployed archive format that stores all kinds
> of file metadata on POSIX systems.  However, multiple options for
> the outer format has been debated.

You mention POSIX, which triggered me.  I think it would be good to
specify which tar format to use.

POSIX.1-2001/pax format doesn't have a 100/256 char filename length
restriction, which is good but it is not (yet) used by default by GNU
tar.  busybox tar can read pax tars, it seems.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format

2018-11-21 Thread Fabian Groffen
reason.  And before you ask really
> > > silly questions, yes, I did fight binary packages over hex editor
> > > at some point.
> > 
> > Which I still don't understand, to be frank.  I think even Portage
> > exposes python APIs to get to the data.
> 
> Compare the time needed to make a trivial (but unforeseen) change
> on a format that's transparent vs a format that requires you to learn
> its spec and/or API, write a program and debug it.

I was under the impression you could unpack a tbz2 into data and xpak,
then unpack both, modify the contents with an editor or whatever, and
then pack the whole stuff back into a tbz2 again.  This can be done
worst case scenario by emerge -k , modifying the vdb and quickpkg
 afterwards.
I know that with portage-utils you can do this easily with the qtbz2 and
qxpak commands.  No need to do anything with a hex editor, or know
anything about how it's done.
Obvious advantage of your approach is that you don't need q* tools, but
can use tar instead.  The editting is as trivial though.  In your case
you need a special procedure to reconstruct the binpkg should you want
to keep your special properties (label, order) which equates to q* tools
somewhat.

> > > The most trivial case is an attempted recovery of a broken system.
> > > If you don't have Portage working and don't have portage-utils
> > > installed, do you really prefer a custom format which will require you
> > > to fetch and compile special tools?  Or is one that can be processed
> > > with tools you're quite likely to have on every system, like tar?
> > 
> > Well, I think the idea behind the original binpkg format was to use tar
> > directly on the files in emergency scenarios like these...
> > The assumption was bzip2 decompressor and tar being available.
> > I think it is an example of how you add something, while still allowing
> > to fallback on existing tools.
> 
> Except progress in compressors has made it work less and less reliably. 
> It's mostly an example how to be *clever*.  However, being clever
> usually doesn't pay off in the long term, compared to doing things *in a
> simple way*.

We agree it is hackish, and we agree we can do without.  You simply
exaggerate the problem, IMO, which mostly isn't there, because it works
fine today.  It can also be solved today using shell tools.

% head -c `grep -abo 'XPAKPACK' 
$EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | sed 's/:.*$//'` 
$EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | tar -jxf -

results in no warnings/errors from bzip about trailing garbage, possible
thanks to the spec being smart enough about this.

Not having to do this, when under stress and pressure to restore a
system to get it back into production, is a plus.  Though, in that
scenario the trailing garbage warning wouldn't have been that bad
either.

> > > > > 3. **The file format should provide for partial fetching of binary
> > > > >packages.**  It should be possible to easily fetch and read
> > > > >the package metadata without having to download the whole package.
> > > > 
> > > > Like above, what is the use-case here?  Why would you want this?  I
> > > > think I'm missing something here.
> > > 
> > > Does this harm anything?  Even if there's little real use for this, is
> > > there any harm in supporting it?  Are we supposed to do things the other
> > > way around with no benefit just because you don't see any real use for
> > > it?
> > 
> > Well, you make a huge point out of it.  And if it isn't used, then why
> > bother so much about it.  Then it just looks like you want to use it as
> > an argument to get rid of something you just don't like.
> > 
> > In my opinion you better just say "hey I would like to implement this
> > binpkg format, because I think it would be easier to support with
> > minimal tools since it doesn't have custom features".  I would have
> > nothing against that.  Simple and elegant is nice, you don't need to
> > invent arguments for that, in my opinion.
> 
> The spec is now more focused on that.

Thank you, much appreciated.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format

2018-11-18 Thread Fabian Groffen
On 18-11-2018 10:38:51 +0100, Michał Górny wrote:
> On Sun, 2018-11-18 at 10:16 +0100, Fabian Groffen wrote:
> > On 17-11-2018 12:21:40 +0100, Michał Górny wrote:
> > > Problems with the current binary package format
> > > ---
> > > 
> > > The following problems were identified with the package format currently
> > > in use:
> > > 
> > > 1. **The packages rely on custom binary archive format to store
> > >metadata.**  It is entirely Gentoo invented, and requires dedicated
> > >tooling to work with it.  In fact, the reference implementation
> > >in Portage does not even include a CLI tool to work with tbz2
> > >packages; an unofficial implementation is provided as part
> > >of portage-utils toolkit [#PORTAGE-UTILS]_.
> > 
> > I think you should rewrite this section to the argument that the
> > metadata is hard to edit, and that there is only one tool to do so
> > (except a python interface from Portage?).
> > On a separate note, I don't think portage-utils can be considered
> > "unofficial", it is a Gentoo official project as far as I am aware.
> 
> In this context, Portage is 'official'.  Portage-utils is a project
> that's developed entirely separately from Portage and doesn't use
> Portage APIs but instead reinvents everything.  As such, it is easy for
> the two to go out of sync.  Or for one of them to have bugs that
> the other one doesn't have (say, with endianness).

I'm not sure if it's actually true, I was under the impression the same
author(s) worked on the Portage as well as portage-utils code.  Anyway,
aren't quickpkg and emerge enough from a user's perspective?

> > > 2. **The format relies on obscure compressor feature of ignoring
> > >trailing garbage**.  While this behavior is traditionally implemented
> > >by many compressors, the original reasons for it have become long
> > >irrelevant and it is not surprising that new compressors do not
> > >support it.  In particular, Portage already hit this problem twice:
> > >once when users replaced bzip2 with parallel-capable pbzip2
> > >implementation [#PBZIP2]_, and the second time when support for zstd
> > >compressor was added [#ZSTD]_.
> > 
> > I think this is actually the result of a rather opportunistic
> > implementation.  The fault is that we chose to use an extension that
> > suggests the file is a regular compressed tarball.
> > When one detects that a file is xpak padded, it is trivial to feed the
> > decompressor just the relevant part of the datastream.  The format
> > itself isn't bad, and doesn't rely on obscure behaviour.
> 
> Except if you don't have the proper tools installed.  In which case
> the 'opportunistic' behavior made it possible to extract the contents
> without special tools... except when it actually happens not to work
> anymore.  Roy's reply indicates that there is actually interest in this
> design feature.

Your point is that the format is broken (== relies on obscure compressor
feature).  My point is that the format simply requires a special tool.
The fact that we prefer to use existing tools doesn't imply in any way
that the format is broken to me.
I think you should rewrite your point to mention that you don't want to
use a tool that doesn't exist in @system (?) to unpack a binpkg.  My
guess is that you could use some head/tail magic in a script if the
trailing block is upsetting the decompressor.

I'm not saying this may look ugly, I'm just saying that your point seems
biased.

> > > 3. **Placing metadata at the end of file makes partial fetches
> > >complex.**  While it is technically possible to obtain package
> > >metadata remotely without fetching the whole package, it usually
> > >requires e.g. 2-3 HTTP requests with rather complex driver.  For
> > >comparison, if metadata was placed at the beginning of the file,
> > >early-terminated pipeline with a single fetch request would suffice.
> > 
> > I think this point needs to be quantified somewhat why it is so
> > important.
> > I may be wrong, but the average binpkg is small, <1MiB, bigger packages
> > are <50MiB.
> > So what is the gain to be saved here?  A "few" MiBs for what operation
> > exactly?  I say "few" because I know for some users this is actually not
> > just a blib before it's downloaded.  So if this is possible to achieve,
> > in what scenarios is this going to be used (and is this often?).
> 
> Last I checked, Gentoo aimed to support more users than the 'majority'
> of peopl

Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format

2018-11-18 Thread Fabian Groffen
 metadata updates.**
>In particular, it should be possible to update the metadata without
>having to recompress package files.
> 
> 6. **The file format should account for easy recognition both through
>filename and through contents.**  Preferably, it should have distinct
>features making it possible to detect it via file(1).
> 
> 7. **The file format should allow for metadata compression.**
> 
> 8. **The file format should make future extensions easily possible
>without breaking backwards compatibility.**

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)

2018-11-12 Thread Fabian Groffen
On 11-11-2018 21:53:33 +0100, Michał Górny wrote:
> Hi,
> 
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
> 
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix.  It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
> 
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
> 
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
> 
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
> 
> 4. "contents.tar${comp}" tarball containing files to be installed.
> 
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
> 
> Notes:
> 
> a. ${comp} can be any compression format supported by binary packages. 
> Technically, metadata and content archives may use different
> compression.  Either or both may be uncompressed as well.

I'm wondering here, how much sense does it make to compress 2., 3.
and/or 4. if you compress the whole gpkg?  I have the impression
compression on compression isn't beneficial here.  Shouldn't just
compressing of the gpkg tar be sufficient?

As to allowing different compressors for a single gpkg, I think it would
be better to require all compressors to be the same, such that a PM or
tool can quickly see if it can "read" the file from the gpkg filename,
instead of having to fetch and open it first.  Obviously, if you drop
compression of the inner tars, this point goes away.

Thanks,
Fabian

> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
> 
> 
> Advantages
> --
> Guaranteed:
> 
> + The binary package is still one file, so can be fetched easily.
> 
> + File format is trivial and can be extracted using tar(1) + compressor.
> 
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
> 
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
> 
> + Metadata can be compressed now.
> 
> Achieved by regular archives (but might be broken if modified by user):
> 
> + Easy recognition by magic(1).
> 
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
> 
> 
> Why not .ar format?
> ---
> The use of .ar format has been proposed, akin to Debian.  While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure.  It lives mostly as static library format, and the tooling
> for it is part of binutils.  LSB considers it deprecated.  While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
> 
> 
> Is there anything left to address?
> 
> -- 
> Best regards,
> Michał Górny



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] app-emulation/libvirt-snmp: version bump to 0.0.4

2018-10-27 Thread Fabian Groffen
On 27-10-2018 15:18:28 +0300, Joonas Niilola wrote:
> Why can't you use Github like _everyone_ else? It's really simple and fast.

Hrm ... given Github's not integrated with Gentoo, one can argue that
sending a patch to the ML is actually more simple/fast for developers.

I don't think sending patches to this ML targets the right audience, yet
I welcome the contribution.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-14 Thread Fabian Groffen
I think you misunderstood what I wrote, or I wasn't clear enough.
Richard summed up my intention nicely in his response.

Fabian

On 15-09-2018 00:46:24 +0300, Alon Bar-Lev wrote:
> On Sat, Sep 15, 2018 at 12:29 AM Fabian Groffen  wrote:
> >
> > On 15-09-2018 00:07:12 +0300, Alon Bar-Lev wrote:
> > > >
> > > > Perhaps, if one persists on going this route, only do this for platforms
> > > > that upstream supports, such that arches which will suffer from this
> > > > (typically ppc, sparc, ...) don't have to be blocked by this.
> > >
> > > Exactly in these cases the -Werror is useful as if upstream expects no
> > > warnings then any warning should block installation and trigger bug
> > > report. In Gentoo in many cases we use packages on platform has no
> > > access to, our feedback to upstream is valuable. A great example is
> > > gnutls in which we collectively (maintainer, unstable users,
> > > architecture teams, stable users) found issues on architectures that
> > > almost nobody other than Gentoo has access to.
> > >
> >
> > I don't believe Gentoo users are (supposed to be) an extension of
> > upstreams.
> 
> This is exactly what I think that is special about Gentoo, and the
> reason I use Gentoo. Unlike other distribution Gentoo is the closest
> thing of using upstream. A maintainer in Gentoo who is not see himself
> part of the upstream packages he maintains has far less impact than a
> maintainer who does see himself as part of upstream or is upstream.
> 
> Per your statement, we should not allow any architecture or setup that
> upstream, such as exact versioning, architecture or toolchain.
> 
> > If upstreams insist on that, they should make their software
> > non-free, adding a non-modification clause or something.  In any case,
> > it is not Gentoo's job IMHO.
> 
> Then we cannot re-distribute or patch, how is it related to the
> discussion? We are talking about open source projects and I know it is
> cliche... the "greater good" and helping the "free open source
> movement" a a viable alternative. I thought this is what unite us
> here.
> 
> > In the end it is Gentoo who needs to care
> > for its users.  I prefer we do that by giving them an option to become
> > that extension of upstream, e.g. by USE=upstream-cflags, which Gentoo
> > disables by default.
> 
> Do you think someone do not care about the users? Do you actually
> think upstream does not care about users? I do not understand this
> statement. If downstream maintainer believes that upstream is friendly
> for the Gentoo overhead (which is higher than binary distributions) or
> create the relationship in which Gentoo is 1st citizen at upstream,
> why do you think users cannot use vanilla upstream?
> 
> > As maintainer and/or enthusiastic user, like you wrote for gnutls, I
> > would be more than happy to provide build logs/errors for all the arches
> > I have access to.  So like I wrote before, I think we should consider
> > case-by-case basis to make it easy to do so.
> 
> This entire discussion is to allow case-by-case and not black and
> white approach recently enforced.
> 
> Regards,
> Alon
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-14 Thread Fabian Groffen
On 15-09-2018 00:07:12 +0300, Alon Bar-Lev wrote:
> >
> > Perhaps, if one persists on going this route, only do this for platforms
> > that upstream supports, such that arches which will suffer from this
> > (typically ppc, sparc, ...) don't have to be blocked by this.
> 
> Exactly in these cases the -Werror is useful as if upstream expects no
> warnings then any warning should block installation and trigger bug
> report. In Gentoo in many cases we use packages on platform has no
> access to, our feedback to upstream is valuable. A great example is
> gnutls in which we collectively (maintainer, unstable users,
> architecture teams, stable users) found issues on architectures that
> almost nobody other than Gentoo has access to.
>

I don't believe Gentoo users are (supposed to be) an extension of
upstreams.  If upstreams insist on that, they should make their software
non-free, adding a non-modification clause or something.  In any case,
it is not Gentoo's job IMHO.  In the end it is Gentoo who needs to care
for its users.  I prefer we do that by giving them an option to become
that extension of upstream, e.g. by USE=upstream-cflags, which Gentoo
disables by default.

As maintainer and/or enthusiastic user, like you wrote for gnutls, I
would be more than happy to provide build logs/errors for all the arches
I have access to.  So like I wrote before, I think we should consider
case-by-case basis to make it easy to do so.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-14 Thread Fabian Groffen
On 14-09-2018 16:29:43 -0400, Rich Freeman wrote:
> On Fri, Sep 14, 2018 at 4:20 PM Michael Orlitzky  wrote:
> >
> > On 09/14/2018 03:58 PM, Richard Yao wrote:
> > >>
> > >> No one has answered the question: what do you do when a stable package
> > >> breaks because of a new warning?
> > >>
> > >> ...>
> > > Wouldn’t this be largely covered as part of GCC stabilization? We could 
> > > reserve the right to kill -Werror in a package where it blocks GCC 
> > > stabilization if the maintainer does not handle it in a timely manner.
> > >>
> >
> > They would be uncovered during GCC stabilization, but then you're right
> > back in the original situation: how do you fix the stable package? The
> > only answer that doesn't violate some other policy is to patch it in a
> > new revision and wait for it to stabilize again.
> >
> > Other questions arise: Do we block stabilization of clang et al.?
> >
> 
> Presumably we could make it a blocker, so then portage won't install
> the new stable toolchain.  That buys time and only affects users of
> that particular package.  But, as I pointed out before you can do that
> without using -Werror - just block installation with an unqualified
> toolchain.
> 
> You would only use an approach like this for packages where QA was
> fairly important, so the inconvenience would be worth it.

Perhaps, if one persists on going this route, only do this for platforms
that upstream supports, such that arches which will suffer from this
(typically ppc, sparc, ...) don't have to be blocked by this.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-13 Thread Fabian Groffen
On 13-09-2018 18:56:13 +0300, Alon Bar-Lev wrote:
> On Thu, Sep 13, 2018 at 6:51 PM Fabian Groffen  wrote:
> >
> > On 12-09-2018 17:46:03 -0700, Matt Turner wrote:
> > > On Wed, Sep 12, 2018 at 5:11 PM Rich Freeman  wrote:
> > > With new GCC comes new warnings, and harmless as the vast majority are
> > > they cause the build to break with Werror.
> >
> > To illustrate harmless:
> >   warning: this statement may fall through [-Wimplicit-fallthrough=]
> > The warning message already has it in it that it's just a pure guess.
> 
> One that exposed a lot of unintentional fallthoughs which were fixed
> when reporting to upstream.

Sure that's why the warning is there.  But you ignore the point that the
same code compiled fine and ran fine for years without problems.

> Once again... we should discuss to leave -Werror when policy of
> upstream to have no warnings and is maintaining that policy properly
> while we at downstream may cooperate and avoid patching upstream but
> discuss issues when found.

On a developer's system, that would be nice.

For ordinary users on the other hand:
Leaving -Werror is leaving our users alone in the dark.  Don't do that.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-13 Thread Fabian Groffen
On 12-09-2018 20:09:54 -0400, Rich Freeman wrote:
> On Wed, Sep 12, 2018 at 7:32 PM Chí-Thanh Christopher Nguyễn
>  wrote:
> >
> > Alon Bar-Lev schrieb:
> > > We
> > > are unique as permutations and architectures that are used by Gentoo
> > > users are so diverse that we find issues that nobody else finds.
> >
> > This needs to be highlighted more, as it is why suggestions that the
> > maintainer can simply put -Werror back on their own system are insufficient.
> >
> 
> Wouldn't running into new runtime issues be exactly the sort of reason
> why we'd want to build with -Werror on packages where these issues are
> unacceptable?

Can you think of a way in which a new runtime issue would occur that
-Werror would have guarded?  And that issue would also get through
maintainer and ~arch testing?

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-13 Thread Fabian Groffen
On 13-09-2018 07:36:09 -0400, Richard Yao wrote:
> 
> 
> > On Sep 12, 2018, at 6:55 PM, Thomas Deutschmann  wrote:
> > 
> >> On 2018-09-12 16:50, Rich Freeman wrote:
> >> There is also the case where we want these warnings to block
> >> installation, because the risk of there being a problem is too great.
> > 
> > I really disagree with that. So many devs have already said multiple
> > times in this thread that "-Werror" is only turning existing warnings
> > into fatal errors but "-Werror" itself doesn't add any new checks and
> > more often requires "-O3" to be useful.
> The way that compilers work is that the warnings are generated in the front 
> end while the optimization level affects the backend. That means that -O3 has 
> no effect on the code that does error generation. This remark about -O3 being 
> needed to make -Werror useful is just plain wrong. 

Huh?  -O3 enables more checks, which can generate more warnings.  -O3
isn't "needed", but if upstream is so interested in clean and correct
code, they should've fixed all warnings in the first place and thus
enabled all of them.  In fact, I expect every sane upstream to use "-O3
-Wall -Werror" in one of their automated builds.  Not that this catches
anything useful on x86{,_64} when there is for instance use of signed
and unsigned char types, so it isn't conclusive.

The whole point in here is that -Werror doesn't add much if you care.
The whole point why it is not desired in Gentoo is that users don't
necessarily are developers, or even interested in fixing warnings --
regardless whether they point to real problems or not.

If there are real problems in a package (exposed by a compiler or not)
then this should ideally stand out during ~arch testing, or even before
when the Gentoo maintainer examines the build (might even use -Werror
for his own purposes).  If such code ends up in stable arch we just made
a stabilisation mistake, or got royally messed up by upstream, depending
how you look at it.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-13 Thread Fabian Groffen
On 12-09-2018 17:46:03 -0700, Matt Turner wrote:
> On Wed, Sep 12, 2018 at 5:11 PM Rich Freeman  wrote:
> >
> > On Wed, Sep 12, 2018 at 7:52 PM Matt Turner  wrote:
> > >
> > > On Wed, Sep 12, 2018 at 4:03 PM Rich Freeman  wrote:
> > > > Now, I could buy that -Werror turns NEW warnings into fatal errors,
> > > > due to the use of a newer toolchain, since upstream probably didn't
> > > > test with that toolchain and thus wouldn't have seen the warning.
> > >
> > > Yes, exactly. This is one of the major things people have said repeatedly.
> > >
> > > Werror makes moving gcc forward much more difficult.
> > >
> >
> > Sure, but wouldn't a block on newer versions of gcc cause similar headaches?
> 
> Sure...? Who is suggesting that? I'm not sure where you're going with this.
> 
> With new GCC comes new warnings, and harmless as the vast majority are
> they cause the build to break with Werror.

To illustrate harmless:
  warning: this statement may fall through [-Wimplicit-fallthrough=]
The warning message already has it in it that it's just a pure guess.


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-13 Thread Fabian Groffen
On 13-09-2018 00:55:45 +0200, Thomas Deutschmann wrote:
> Unless you can do that we don't really need to discuss this. Simply
> because everyone interested in "-Werror" *can* set that flag via CFLAGS,
> even just per package, whereas the majority, not interested in this,
> cannot do the same to filter "-Werror". Nobody advocating for "-Werror"
> replied to that fact yet.

Unfortunately adding -Werror to CFLAGS is not just possible.  Many
configure checks rely on the compiler just generating something,
ignoring warnings for various reasons.  If you add -Werror to your
CFLAGS you basically introduce breakage for some autoconf-based
packages.

What we really should be having is an easy way for post-configure CFLAGS
addition.  Just to support devs/users who insist on this for their
setups.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Changing policy about -Werror

2018-09-10 Thread Fabian Groffen
On 09-09-2018 11:22:41 -0400, Richard Yao wrote:
> -Werror has caught bugs that could have resulted in data loss in ZFS in the 
> past thanks to it being built in userspace as part of zdb. So it is useful 
> for integrity too, not just security (although arguably, integrity is part of 
> security).

This is a misconception, as jer already pointed out.  Instead:

-Werror has forced you to take notice of problems that could have
resulted in data loss in ZFS ...

Also, consider that for -Werror to be "better", you also need -O3 in
order to activate the "proper" compiler checks like "variable set but
never used" ones.

> Perhaps we could have another USE flag for -Werror where it is a security 
> feature. e.g. USE=strict-compile-checks

You better run a static code analyser, such as the one you can hook up
with Travis.  It usually points out real security problems such as
races, which GCC doesn't do yet, as far as I'm aware.  Let alone
trigger with -Werror.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] multilib: allow specifying the subtype of library in get_libname

2018-08-03 Thread Fabian Groffen
Just wondering, we have get_libname, get_modname, shouldn't we just
introduce get_linklibname or something instead of trying to overload
get_libname?

The implementation of get_linklibname could be to just call get_libname
for anything but mingw of course.

Fabian

On 03-08-2018 00:09:41 -0500, Marty E. Plummer wrote:
> Signed-off-by: Marty E. Plummer 
> ---
> 
> On mingw-w64 (and perhaps cygwin and mingw.org), there are two forms of
> non-static libraries. Standard *.dll libraries are for runtime and are
> loaded from %PATH% on windows systems, and are typically stored in
> either /bin or /usr/bin on mingw-w64 cross-toolchain filesystems. Import
> libraries, *.dll.a, are used when linking and live in the ''normal''
> libdirs, eg, /lib, /usr/lib and so on.
> 
> A number of ebuilds which otherwise work on mingw-w64 crossdev
> toolchains exhibit failure due to usage of get_libname not being able to
> specify which of the two types are required.
> 
> For example, sys-libs/ncurses, uses the following snippet of code:
> ln -sf libncurses$(get_libname) 
> "${ED}"/usr/$(get_libdir)/libcurses$(get_libname) || die
> in order to create a 'libcurses.so -> libncurses.so' symlink.
> 
> However, on a crossdev-built mingw-w64 toolchain, one will end up with a
> broken 'libcurses.dll -> libncurses.dll' symlink, which should properly
> be a 'libcurses.dll.a -> libncurses.dll.a' symlink, as the symlink here
> is provided to allow linking with -lcurses instead of -lncurses.
> 
>  eclass/multilib.eclass | 52 ++
>  1 file changed, 48 insertions(+), 4 deletions(-)
> 
> diff --git a/eclass/multilib.eclass b/eclass/multilib.eclass
> index 350b6f949d1..6a99f5977ec 100644
> --- a/eclass/multilib.eclass
> +++ b/eclass/multilib.eclass
> @@ -239,26 +239,70 @@ get_exeext() {
>  }
>  
>  # @FUNCTION: get_libname
> -# @USAGE: [version]
> +# @USAGE: --link|--run [version]
>  # @DESCRIPTION:
>  # Returns libname with proper suffix {.so,.dylib,.dll,etc} and optionally
>  # supplied version for the current platform identified by CHOST.
>  #
> +# If '--link' argument is passed, the linktime library's suffix is returned,
> +# as in the file that must exist to let `gcc -lfoo foo.c -o foo` to work.
> +# If '--run' argument is passed, the runtime library's suffix is returned.
> +#
> +# In most unix-like platforms the two are identical, however on mingw-w64 the
> +# linktime library has the suffix of '.dll.a' and the runtime library '.dll'.
> +#
>  # Example:
>  # get_libname ${PV}
>  # Returns: .so.${PV} (ELF) || .${PV}.dylib (MACH) || ...
> +# get_libname --link
> +# Returns: .so (ELF) || .dylib (MACH) || .dll.a (PE32) || ...
>  get_libname() {
> - local libname
> - local ver=$1
> + local libtype="undefined"
> + local libname opt ver
> + for opt; do
> + case "${opt}" in
> + --link)
> + libtype="link"
> + shift
> + ;;
> + --run)
> + libtype="run"
> + shift
> + ;;
> + *)
> + ;;
> + esac
> + done
> + ver="$1"
> + # general unixy types
>   case ${CHOST} in
>   *-cygwin*)   libname="dll.a";; # import lib
> - mingw*|*-mingw*) libname="dll";;
>   *-darwin*)   libname="dylib";;
>   *-mint*) libname="irrelevant";;
>   hppa*-hpux*) libname="sl";;
>   *)   libname="so";;
>   esac
>  
> + # wierd mingw-w64 stuff, maybe even cygwin
> + case ${CHOST} in
> + mingw*|*-mingw*)
> + case ${libtype} in
> + link)
> + libname="dll.a" # import library
> + ;;
> + run)
> + libname="dll" # runtime library
> + ;;
> + undefined)
> + eerror "please specify either --link or 
> --run to get_libname"
> + eerror "for mingw builds, as there are 
> two types of libraries"
> + eerror "on this platform"
> + die
> + ;;
> + esac
> + ;;
> + esac
> +
>   if [[ -z $* ]] ; then
>   echo ".${libname}"
>   else
> -- 
> 2.18.0
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] rfc: [QA] Ban policy introduction

2018-07-29 Thread Fabian Groffen
Completely agreeing with Sergei, with some additional suggestions:

On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
> On Sun, 29 Jul 2018 00:40:18 +0300
> Mikle Kolyada  wrote:
> 
> > Hello,
> > 
> > The Gentoo QA team would like to introduce the following policy that
> > would be applied to individuals breaking the state and quality of the
> > main gentoo.git tree
> > 
> > ( as we do not have this strictly documented yet):
> > 
> > 
> > 
> > If recommended
> 
> It's not called "recommended" but "enforced".

I agree.  If you put penalties on these, they become hard rules.  I
think that change should be discussed by the council perhaps?

> > Gentoo workflow policies are not followed by an
> > individual developer
> > (e.g make major changes to the widely used eclasses without prior
> > discussion on the mailing list or
> > commit changes that lead to multiple CI checks failure)
> 
> Here should go exhaustive list of links to the policies to be enforced.

At least.  And they should be clear and concise.  No "common sense" or
anything involved for exceptions and the like.  In addition, new checks
should be introduced to the community and possibly approved by council
as to whether being enforced or not.

Fabian

> 
> > the standard QA
> > procedure is:
> > 
> > 1.) Two warnings granted by QA team, after two independent breakages
> > 2.) Revoking the commit access for 14 days
> > 
> > These violations will be evaluated individually by all QA team members.
> > Warnings can be revoked, if during 6 months period a developer makes at
> > least 20 non trivial changes not producing more breakages.
> > 
> > 
> 
> -- 
> 
>   Sergei
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)

2018-07-08 Thread Fabian Groffen
lg = False
>   is_synced = False
>   if timestamp != 0 and "--quiet" not in opts:
> @@ -686,6 +752,12 @@ class RsyncSync(NewBase):
>   elif (servertimestamp == 0) or (servertimestamp > 
> timestamp):
>   # actual sync
>   command = rsynccommand[:]
> +
> + if self.repo.location != download_dir:
> + # Use shared hardlinks for files that 
> are identical
> + # in the previous snapshot of the 
> repository.
> + command.append('--link-dest=%s' % 
> self.repo.location)
> +
>   submodule_paths = self._get_submodule_paths()
>   if submodule_paths:
>   # The only way to select multiple 
> directories to
> @@ -696,9 +768,10 @@ class RsyncSync(NewBase):
>   # /./ is special syntax 
> supported with the
>   # rsync --relative option.
>   command.append(syncuri + "/./" 
> + path)
> - command.append(self.repo.location)
>   else:
> - command.extend([syncuri + "/", 
> self.repo.location])
> + command.append(syncuri + "/")
> +
> + command.append(download_dir)
>  
>   exitcode = None
>   try:
> -- 
> 2.13.6
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH v2 09/11] glep-0063: Make recommended expiration terms mandatory

2018-07-06 Thread Fabian Groffen
On 06-07-2018 13:34:21 +0200, Ulrich Mueller wrote:
> - Make creation of a revocation certificate (and storing it in a place
>   separate from the key) mandatory.

What does this really achieve?  Or require?  Am I supposed to buy or
hire a vault now?  --  I'm assuming the word "safe" is missing from
above sentence.

Side observation:
You can't check I have the revocation cert, let alone that you can
check where it is stored, or if I lost it.

Unless it is stored in a Gentoo owned vault or something, such that
infra can invoke it on retirement scripts, this seems like unnecessary
bureaucracy.

Of course we want to encourage anyone to have a revocation cert, and to
store it in a safe place somewhere.  This is at best subject to means
and opportunities of the person in question.  In reality it is quite
hard to store secrets securely, even more when they don't fit well in
the human SSD.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Developer commit timeline updates

2018-06-25 Thread Fabian Groffen
On 24-06-2018 00:36:28 +0200, Michał Górny wrote:
> I'd like to just make a short announcement that I've updated
> the developer commit timeline [1].

Very nice stats, thanks for that.

Suggestion: perhaps this can be cron-jobbed and updated every month or
so?

In any case, fun to look at. Thanks :)


> [1]:https://dev.gentoo.org/~mgorny/dev-timeline.html
> [2]:https://dev.gentoo.org/~mgorny/active-devs.html
-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Fabian Groffen
On 01-04-2018 20:29:59 +0200, Michał Górny wrote:
> > > This essentially looks like ~700 lines of code to try to workaround
> > > broken networking. I would rather try to do that using 5 lines of code
> > > but that's just me, and my programs aren't enterprise quality. I just
> > > hope it actually solves as many problems as it's going to introduce.
> > 
> > The vast majority of this code is generic and reusable, and I do intend
> > to reuse it. For example, the executor support will be an essential
> > piece for the asyncio.AbstractEventLoop for bug 649588.
> 
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.

Well, well, well.  This could be said about a lot of code, including
your own.

> Last time you went away, you left us with a horribly unmaintainable
> package manager full of complexity, hacks and creeping featurism
> and a Portage team whose members had barely any knowledge of the code.
> Just when things started moving again, you came back and we're back to
> square one.

I don't see why this has to be made personal.  In the olde days people
just pushed whatever they needed, now it's reviewed and acked, so how
can it be back to square one?

> Today Portage once again is a one-developer project, full of more
> complexity, more hacks and more creeping featurism. And we once again
> have a bus factor of one -- one developer who apparently knows
> everything, does everything and tries to be nice to everyone, except he
> really ignores others, makes a lot of empty promises and consistently
> makes the health of the project go from bad to worse.

Errr, didn't you recently start your own fork with creeping featurism of
its own because mainline was/is too slow?

> So, please tell me: what happens when you leave again? How have you used
> your time in the project? What have you done to make sure that
> the project stays alive without you? Because as far as I can see, adding
> few thousand of lines of practically unreviewed code every month does
> not help with that.

Why is this Zac's problem specifically?  Isn't this a general problem of
Gentoo?  And isn't your attitude contributing in a large factor to
the bus-factor getting down from 1 to 0?

> I forked Portage because I didn't want to fight with you. When I forked
> it, I declared that I will merge mainline changes regularly for
> the benefit of my users. But after a week, I really start feeling like
> that's going to be a really bad idea. Like it's time to forget about
> mainline Portage as a completely dead end, and go our separate ways.

So you fork.  Like many forks, that is because people feel that it isn't
going "their way".  Yay.  That doesn't make you "right".

> I'm seriously worried about the future of Gentoo. I'd really appreciate
> if you started focusing on that as well. I get that all this stuff looks
> cool on paper but few months or years from now, someone will curse
> 'whoever wrote that code' while trying to fix some nasty bug. Or get
> things moving forward. Or implement EAPI 8.

Perhaps it's time to look into the mirror.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] rfc: empty directories in ${D}

2018-03-29 Thread Fabian Groffen
On 29-03-2018 16:47:51 +0200, Michał Górny wrote:
> W dniu czw, 29.03.2018 o godzinie 09∶39 -0500, użytkownik William Hubbs
> napisał:
> > All,
> > 
> > I just happened to notice the following warning from portage when
> > bumping dhcpcd.
> > 
> > > One or more empty directories installed to /var:
> > >  /var/lib/dhcpcd
> > > If those directories need to be preserved, please make sure to create
> > > or mark them for keeping using 'keepdir'. Future versions of Portage
> > > will strip empty directories from installation image.
> > 
> > If we are going to require emptty  directories to be marked with
> > keepdir, I think we should hard fail the emerge rather than quietly
> > strip the empty directories. If we just strip the directories, this
> > will, more than likely, lead to broken packages. In the case of dhcpcd,
> > the upstream build system installs the /var/lib/dhcpcd directory, then
> > dhcpcd writes to the directory.
> > 
> 
> Are you saying that dozens of packages should suddenly start failing
> for users so that developers would feel more obliged to fix them?
> Provided that the packages are still maintained, and it won't be
> 'hey, we just made it impossible to install this package, maybe someone
> will fix it one day'.

I agree, packages shouldn't suddenly start failing.  Not during install,
not during runtime either.  For changes like this EAPIs were invented.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New category: app-metrics

2018-03-22 Thread Fabian Groffen
Hi!

On 20-03-2018 06:21:00 +0100, Manuel Rüger wrote:
> Hi everyone,
> 
> I'm planning to add a new category app-metrics, which is mainly (at
> least for my own use case) going to be used for prometheus[0] and its
> exporters providing endpoints for prometheus. It can be used for other
> packages whose _main_ purpose is to provide metrics, transform or
> consume them.
> 
> * net-analyzer/prometheus
> * app-admin/bind_exporter
> * app-admin/elasticsearch_exporter
> * app-admin/mongodb_exporter
> * app-admin/nginx-vts-exporter
> * app-admin/prom2json
> * dev-util/buildbot-prometheus

Does the graphite stack fit in this too, you think?

app-admin/diamond
app-admin/collectd
app-misc/carbon-c-relay
dev-python/carbon
net-analyzer/graphite-web
www-apps/grafana-bin

We might have more packages, but this is from the top off my head.

Thanks,
Fabian

> 
> In addition, the following packages will drop their prometheus- prefix
> during the package move:
> 
> * net-analyzer/prometheus-blackbox_exporter
> * net-analyzer/prometheus-node_exporter
> * net-analyzer/prometheus-redis_exporter
> * net-analyzer/prometheus-snmp_exporter
> * net-analyzer/prometheus-uwsgi_exporter
> * net-analyzer/prometheus-pushgateway
> * net-analyzer/prometheus-alertmanager
> 
> With the growing adoption of prometheus I expect more exporters to be
> added (I have five more that I want to add in the near future).
> 
> 
> Thanks,
> 
> Manuel
> 
> [0] https://prometheus.io
> 
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] About making mastersync redundant

2018-03-20 Thread Fabian Groffen
Hi,

I know infra is working on fixing this, so they better focus on that for
now.  Thank you to infra for doing all the work!

When this is resolved, perhaps we should have a discussion on how to
make this service redundant?  Currently the prefix rsync generation is
redundant (== 2 generators) so I'm interested to discuss and see if that
would be feasible for gx86 too.

Fabian

On 19-03-2018 21:13:37 -0400, Alec Warner wrote:
> This is just an FYI: [1]https://infra-status.gentoo.org/
> 
> Hoping to have this fixed in a couple of days. In the meantime you may see
> missing snapshots (for emerge webrsync) and no rsync propagation.
> 
> -A
> 
> 
> 
>  References:
>1. https://infra-status.gentoo.org/
> 
> read_char: errno==EILSEQ; invalid byte sequence for UTF-8: 
-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] about non-ebuild files in the tree (and verification thereof)

2018-02-28 Thread Fabian Groffen
On 28-02-2018 22:08:54 +, Robin H. Johnson wrote:
> On Wed, Feb 28, 2018 at 04:10:52PM +0100, Fabian Groffen wrote:
> > Hi,
> > 
> > I'm working on a verification implementation of
> > https://www.gentoo.org/glep/glep-0074.html and ran into the following
> > scenario that I don't know if it's right or wrong:
> ...
> > Does anybody know or have a pointer to what the policies on files in our
> > ebuild dirs actually is?
> PMS, 4.3 Package directories:
> A package directory may contain other files or directories, whose
> purpose is not covered by this specification.

Ah, forwards compatibility.

> GLEP74 itself makes no determination of files being permitted in a given
> directory.
> 
> > Now in a rsync checkout of the Prefix tree, where my own implementation
> > also runs the fat manifest creation, this entry is not present, because
> > I always believed only metadata.xml, ChangeLog* and *.ebuild files were
> > allowed.
> I'd say your separate implementation is wrong in this case, but that
> file also should not permit at this time.

I might change it not to bother about what should be in/out, but just
assume it's right as-is.  For now it is a nice headsup about something
being unexpected.

> > Now I'm confused as to whether this is the case or not, I can't find a
> > GLEP or anything, but repoman also is as happy as it can be on this odd
> > file (I thought it used to complain about stray/unadded files).
> I personally think repoman should complain about it because it's weird.

I'm sure this particular file was a mistake, that went unnoticed for a
very long time.  I do feel this should one way or the other not be
allowed.

Thanks for your insights,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] about non-ebuild files in the tree (and verification thereof)

2018-02-28 Thread Fabian Groffen
Hi,

I'm working on a verification implementation of
https://www.gentoo.org/glep/glep-0074.html and ran into the following
scenario that I don't know if it's right or wrong:

Consider net-misc/srf-ip-conn-srv
% ls
files  Manifest  metadata.xml  srf-ip-conn-srv-.ebuild
srf-ip-conn-srv.pid
% cat Manifest
DIST jsmn-35086597a72d.tar.gz 11056 
DIST srf-ip-conn-140c9b8a8619.tar.gz 112882 

Notice that there is a .pid file in the ebuild dir, checked in git,
which even contains what appears to be a pid.  It isn't used from the
ebuild as far as I can tell.  Apart from being odd, this is actually
irrelevant.

In an rsync checkout of the gentoo-x86 tree, I see in the Manifest for
this package a DATA entry for the .pid-file.  Hence, verification with
both gemato as well as my own implementation succeed because the
.pid-file is acknowledged.

Now in a rsync checkout of the Prefix tree, where my own implementation
also runs the fat manifest creation, this entry is not present, because
I always believed only metadata.xml, ChangeLog* and *.ebuild files were
allowed.

Now I'm confused as to whether this is the case or not, I can't find a
GLEP or anything, but repoman also is as happy as it can be on this odd
file (I thought it used to complain about stray/unadded files).

Does anybody know or have a pointer to what the policies on files in our
ebuild dirs actually is?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] about commits in the future

2018-02-21 Thread Fabian Groffen
Please consider commit 2fb923d758749be609c1daab2a72ad4f1ec4c2a9
(use something like
   git log --pretty=fuller app-emulation/nemu/nemu-1.4.0.ebuild)

It was made roughly one day in the future.  I'm wondering:
1) is this bad at all?
2) if it is bad (got me confused for another reason) is it technically
   possible to reject such commits?
3) if 2) should we decide on some clock skew and reject anything which
   is beyond that?

How do others feel about this?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] RFC: Bugzilla arch list reordering

2018-01-23 Thread Fabian Groffen
On 23-01-2018 12:49:00 +0100, Dirkjan Ochtman wrote:
> On Tue, Jan 23, 2018 at 12:46 PM, Michał Górny <[1]mgo...@gentoo.org> wrote:
> 
> > Since neither of the proposals has received any specific reply, I'm not
> > sure how to proceed from here. I suppose we can possibly have two lists
> > in different order so that people could use whichever they prefer.
> 
> Not sure having two lists in Bugzilla would be an improvement. It would be 
> nice
> if more people expressed a preference, though!

I like the "popularity" order, but on top of that, I'd like to group
32/64 bits versions of arches.  Basically djc converted into:

AMD64
X86
PPC64
PPC
ARM64
ARM
SPARC64 (?)
SPARC
ALPHA
IA64
HPPA
MIPS
...

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Re: repo/gentoo:master commit in: sys-devel/llvm/

2018-01-20 Thread Fabian Groffen
On 19-01-2018 22:21:55 +, Michał Górny wrote:
> URL:https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5bcc52f0
> 
> sys-devel/llvm: Fix implicit dependency on app-arch/libxar
> 
> Support conditionally using app-arch/libxar in LLVM 6+, and explicitly

libxar

> @@ -45,6 +45,7 @@ RDEPEND="
>   libedit? ( dev-libs/libedit:0=[${MULTILIB_USEDEP}] )
>   libffi? ( >=virtual/libffi-3.0.13-r1:0=[${MULTILIB_USEDEP}] )
>   ncurses? ( >=sys-libs/ncurses-5.9-r3:0=[${MULTILIB_USEDEP}] )
> + xar? ( app-arch/xar )

xar here

> --- a/sys-devel/llvm/metadata.xml
> +++ b/sys-devel/llvm/metadata.xml
> @@ -20,5 +20,7 @@
>   Support querying terminal properties using 
> ncurses' terminfo
>   Build compiler-rt's sanitizers
>   Install the Clang static analyzer 
> (requires USE=clang)
> + Support dumping LLVM bitcode sections in 
> Mach-O files
> + (uses app-arch/libxar)

I think you mean app-arch/xar everywhere?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] about stable, dev and exp profile status

2018-01-10 Thread Fabian Groffen
On 07-01-2018 21:25:28 +0100, Michał Górny wrote:
> I'd like to follow this with a more precise proposal. Namely, redefine
> the current profile statuses to apply the following:
> 
> a. stable -> fully tested, all depgraph breakages are errors,
> 
> b. exp -> fully tested, all depgraph breakages are warnings,
> 
> c. dev -> developer's playground, not tested.

I always was under the impression the following order (and explanation)
was the case:

stable -> development -> experimental

For this reason, e.g. Prefix profiles are (still) experimental, which
means they really shouldn't bother non-Prefix people.  Prefix users and
developers work on an environment where those profiles are promoted to
development ones, such that repoman kicks in for their work.  (At least
that was the idea.)

I see you (re-)define dev as "developer's playground", and I wonder if
in that case it wouldn't be better to introduce a new one instead?

Maybe I'm just one of a few who thinks the order is reversed now.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Upcoming posting restrictions on the gentoo-dev mailing list

2018-01-10 Thread Fabian Groffen
On 09-01-2018 22:20:56 +0100, Andreas K. Huettel wrote:
> If, as a non-developer, you want to participate in a discussion on 
> gentoo-dev, 
> - either reply directly to the author of a list mail and ask him/her to 
> forward your message,
> - or ask any Gentoo developer of your choice to get you whitelisted.
>
> If, as a developer, you want to have someone whitelisted, please comment on
> bug 644070 [3]. Similar to Bugzilla editbugs permission, if you are vouching
> for a contributor you are expected to keep an eye on their activity.

For the record, I do not like this decision.  Not just because it puts a
burden on developers to become comrel agents.  A technical solution like
this, doesn't solve the actual problem.  Unfortunately this solution
destroys much more than it intends to fix, which is a loss for Gentoo.

Since this turns -dev into -core-light, I suggest to anyone who doesn't
like this direction that we move discussions to -project instead.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Re: repo/gentoo:master commit in: dev-libs/libunibreak/

2017-12-14 Thread Fabian Groffen
On 14-12-2017 14:41:17 +0100, Michał Górny wrote:
> W dniu czw, 14.12.2017 o godzinie 13∶56 +0100, użytkownik Fabian Groffen
> napisał:
> > On 14-12-2017 13:39:18 +0100, Michał Górny wrote:
> > > Dnia 14 grudnia 2017 13:21:47 CET, Fabian Groffen <grob...@gentoo.org> 
> > > napisał(a):
> > > > Can we make it a policy to list /what/ QA issues are the justification
> > > > for commits like these?  A description in the commit message would be
> > > > preferred, but a pointer to a location where said issues can be found
> > > > would do too.
> > > 
> > > Maintainer-needed is reason enough. If somebody couldn't be bothered to 
> > > maintain what he committed, why should we bother to list the issues?
> > > 
> > > Using repoman and looking at CI mails is also a good idea.
> > 
> > Obviously for me to learn something, I won't/can't use repoman here.  So
> > a pointer to said CI mails from the message of the QA commit would be
> > nice.
> 
> Last I checked, I wasn't personally responsible for teaching people
> ebuild writing 101 while on phone. But here you go (in malformed paste
> of ebuild below).

You simply replied, and therefore took ownership from QA point of view.
I can't help it you do that whilst on the phone.  In fact, this is
email, so being on the phone is not a good reason to be vague and avoid
answering questions in the first place.

[snip issues]

Thanks, much appreciated.  I'm completely convinced now.  I'm referring
back to my earlier suggestion to include such list or the type of issues
found when a drastic commit like the one we discuss is done under the QA
flag.  It's good to know that the QA issue complaint was valid, and
improvements can be made.

A final suggestion is to talk to the committer before taking such
drastic actions, in a situation where our users aren't endangered.  As
this one is, in my opinion.
There is more problematic stuff in the tree, but teach a man to fish ...
next time the problems may be avoided.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-dev] Re: repo/gentoo:master commit in: dev-libs/libunibreak/

2017-12-14 Thread Fabian Groffen
On 14-12-2017 14:43:11 +0100, Michał Górny wrote:
> Bull-shit.
> 
> Breaking the dependency tree was a *honest* mistake on the person who
> reverted this commit.

Honestly, so making mistakes is evaluated based on honesty, then still,
did Andreas (not a QA member as far as I can see) contact Andrey upfront
to establish how honest his mistake was?

> Andrey pretty clearly stated that he did this *on purpose*.

Andreas also did his commit *on purpose*.  Honestly.  And he made things
worse, now actually *affecting* our users.  ...

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-dev] Re: repo/gentoo:master commit in: dev-libs/libunibreak/

2017-12-14 Thread Fabian Groffen
On 14-12-2017 14:34:51 +0100, Michał Górny wrote:
> W dniu czw, 14.12.2017 o godzinie 20∶24 +0700, użytkownik
> gro...@gentoo.org napisał:
> > If the developers of liblinebreak had not decided to rename their library, 
> > I could safely bump it from 2.1 to 4.0, in spite of the fact that it is 
> > maintainer-needed, right?
> > Am I personally responsible for their decision to use the new name 
> > libunibreak?
> 
> No, it wouldn't be fine. We might not even have noticed it if you
> at least bothered to update the Manifest. Please stop looking for
> excuses and loopholes, and start doing something good for Gentoo.

So, breaking the tree, just because someone forgot to set the
maintainer field is doing something good for Gentoo?  (That's called QA?)
https://archives.gentoo.org/gentoo-commits/message/7f34111f0e45f451d5b3a5a58b057d51

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-dev] Re: repo/gentoo:master commit in: dev-libs/libunibreak/

2017-12-14 Thread Fabian Groffen
On 14-12-2017 13:39:18 +0100, Michał Górny wrote:
> Dnia 14 grudnia 2017 13:21:47 CET, Fabian Groffen <grob...@gentoo.org> 
> napisał(a):
> >Can we make it a policy to list /what/ QA issues are the justification
> >for commits like these?  A description in the commit message would be
> >preferred, but a pointer to a location where said issues can be found
> >would do too.
> 
> Maintainer-needed is reason enough. If somebody couldn't be bothered to 
> maintain what he committed, why should we bother to list the issues?

It seems to me you are avoiding the question.  There are no issues with
the ebuild.  It seems like there is just a false claim there are QA
issues, and that is used as waiver to remove the package.

> Using repoman and looking at CI mails is also a good idea.

repoman full (stable) is happy on 8b4ea0f6d2bed140116f69855d1d3100ea0cf020.
qa-reports.gentoo.org has nothing to report
gentoo-qa@ ML has nothing to report

Please list the QA issues:

> >On 14-12-2017 12:10:59 +, Andreas Hüttel wrote:
> >> Also other QA issues.

Apart from that maintainer-needed has nothing to do with Quality of an
ebuild, you mentioned it as an QA issue, so I am interested in the
"other" QA issues, which seems to suggest 2+ problems in this *ebuild*.

For the record, I didn't commit this ebuild.  I'm just extremely unhappy
about the tiggerhippy response of QA which in my opinion is totally
uncalled for, and am extremely worried about the integrity of QA because
of seemingly false claims to justify actions.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


Re: [gentoo-dev] Re: repo/gentoo:master commit in: dev-libs/libunibreak/

2017-12-14 Thread Fabian Groffen
On 14-12-2017 13:39:18 +0100, Michał Górny wrote:
> Dnia 14 grudnia 2017 13:21:47 CET, Fabian Groffen <grob...@gentoo.org> 
> napisał(a):
> >Can we make it a policy to list /what/ QA issues are the justification
> >for commits like these?  A description in the commit message would be
> >preferred, but a pointer to a location where said issues can be found
> >would do too.
> 
> Maintainer-needed is reason enough. If somebody couldn't be bothered to 
> maintain what he committed, why should we bother to list the issues?
> 
> Using repoman and looking at CI mails is also a good idea.

Obviously for me to learn something, I won't/can't use repoman here.  So
a pointer to said CI mails from the message of the QA commit would be
nice.

Thanks,
Fabian

> >
> >Thanks,
> >Fabian
> >
> >On 14-12-2017 12:10:59 +, Andreas Hüttel wrote:
> >> URL:   
> >https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=34e2c43f
> >> 
> >> Revert "dev-libs/libunibreak: initial import"
> >> 
> >> Please write on the chalkboard 100 times: "I will not add ebuilds as
> >maintainer-needed".
> >> Also other QA issues.
> >> This reverts commit 8b4ea0f6d2bed140116f69855d1d3100ea0cf020.
> >> 
> >>  dev-libs/libunibreak/Manifest   |  1 -
> >>  dev-libs/libunibreak/libunibreak-4.0.ebuild | 39
> >-
> >>  dev-libs/libunibreak/metadata.xml   | 14 ---
> >>  3 files changed, 54 deletions(-)
> >> 
> >> diff --git a/dev-libs/libunibreak/Manifest
> >b/dev-libs/libunibreak/Manifest
> >> deleted file mode 100644
> >> index 487fd898f5d..000
> >> --- a/dev-libs/libunibreak/Manifest
> >> +++ /dev/null
> >> @@ -1 +0,0 @@
> >> -DIST libunibreak-4.0.tar.gz 629403 SHA256
> >f7329bef1eb169d3363f040cefcc323cfd0a0bc53290a35a395e1d1adc849539 SHA512
> >43da73f66fabd8fdef444c5a06ad1800464a0aeab590938522d6c19973950a242f2ccc0575a93d10d87bdcf82610452117ac081ddb73f47271a8c2a65897e11c
> >WHIRLPOOL
> >ad71bc5910ca3dff994651022a5a6c6093cd4023852270fa243848f7576287b7cec4ad02a6b27686149491cb52824a93fdb6a6dac4c878d67db2f4041282d300
> >> 
> >> diff --git a/dev-libs/libunibreak/libunibreak-4.0.ebuild
> >b/dev-libs/libunibreak/libunibreak-4.0.ebuild
> >> deleted file mode 100644
> >> index f531bb66e38..000
> >> --- a/dev-libs/libunibreak/libunibreak-4.0.ebuild
> >> +++ /dev/null
> >> @@ -1,39 +0,0 @@
> >> -# Copyright 1999-2017 Gentoo Foundation
> >> -# Distributed under the terms of the GNU General Public License v2
> >> -
> >> -EAPI=6
> >> -inherit versionator
> >> -
> >> -DESCRIPTION="Line and word breaking library"
> >> -HOMEPAGE="http://vimgadgets.sourceforge.net/libunibreak/;
> >>
> >-SRC_URI="https://github.com/adah1972/${PN}/releases/download/${PN}_$(replace_all_version_separators
> >'_')/${P}.tar.gz"
> >> -
> >> -LICENSE="ZLIB"
> >> -SLOT="0"
> >> -KEYWORDS="~amd64 ~arm ~ppc ~x86"
> >> -IUSE="doc +man static-libs"
> >> -
> >> -DEPEND="man? ( app-doc/doxygen )"
> >> -RDEPEND="!dev-libs/liblinebreak"
> >> -
> >> -src_prepare() {
> >> -  use man && echo -e 'GENERATE_MAN=YES\nGENERATE_HTML=NO' >> Doxyfile
> >> -  default
> >> -}
> >> -
> >> -src_configure() {
> >> -  econf \
> >> -  $(use_enable static-libs static)
> >> -  use doc && export HTML_DOCS=( doc/html/. )
> >> -}
> >> -
> >> -src_compile() {
> >> -  default
> >> -  use man && doxygen || die
> >> -}
> >> -
> >> -src_install() {
> >> -  default
> >> -  prune_libtool_files
> >> -  use man && find "${D}/usr/include" -type f -execdir doman
> >"${S}/doc/man/man3/{}.3" \;
> >> -}
> >> 
> >> diff --git a/dev-libs/libunibreak/metadata.xml
> >b/dev-libs/libunibreak/metadata.xml
> >> deleted file mode 100644
> >> index a8bbb441f29..000
> >> --- a/dev-libs/libunibreak/metadata.xml
> >> +++ /dev/null
> >> @@ -1,14 +0,0 @@
> >> -
> >> - >"http://www.gentoo.org/dtd/metadata.dtd;>
> >> -
> >> -  
> >> -  
> >> -  Libunibreak is an implementation of the line breaking and word
> >breaking algorithms
> >> -  as described in Unicode Standard Annex 14 and Unicode Standard
> >Annex 29. It is
> >> -  designed to be used in a generic text renderer.
> >> -  
> >> -  
> >> -  Generate man pages with doxygen.
> >> -  Install html API documentation.
> >> -  
> >> -
> >> 
> >> 
> 
> 
> -- 
> Best regards,
> Michał Górny (by phone)
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: Digital signature


  1   2   3   4   5   6   >