Re: [gentoo-dev] package.use.mask / package.use.stable.mask priority

[Ulrich Mueller]

> On Tue, 10 Jan 2017, Zac Medico wrote:

> On 01/10/2017 01:56 PM, Bernard Cafarelli wrote:
>> gnustep-base/gnustep-make has a USE flag (libobjc2) masked globally in
>> base/package.use.mask, and unmasked on specific arches in
>> arch/{amd64,x86}/package.use.mask
>> 
>> To get a stabilization (#579232) bug finally moving on, I wanted to
>> leave this flag out, adding a corresponding line in
>> base/package.use.stable.mask

So do I understand this correctly, there is:
flag in base/package.use.mask,
-flag in arch/{amd64,x86}/package.use.mask, and
flag in base/package.use.stable.mask?

>> But repoman replied with a batch of dependency.bad errors...
>> Does package.use.mask (stable and ~arch) have a higher priority on
>> package.use.stable.mask (stable only)? Bug or intended behavior?

> If I understand you correctly, then it's the intended behavior. If the
> flag is masked in both package.use.mask and package.use.stable.mask,
> then the package.use.stable.mask setting is irrelevant because both
> package.use.mask and package.use.stable.mask are considered when
> calculating use.mask settings for any given package.

I believe this is not correct. package.use.stable.mask should take
precedence within the same profile:
https://projects.gentoo.org/pms/6/pms.html#x1-58002r1

The problem here is rather that the base profile is processed as a
whole before the arch specific profile, so you end up with -flag from
the arch profile.

Putting flag in arch/{amd64,x86}/package.use.stable.mask should solve
it.

Ulrich


pgpQ1Sv9jeibf.pgp
Description: PGP signature

Re: [gentoo-dev] package.use.mask / package.use.stable.mask priority

[Bernard Cafarelli]

Le Tue, 10 Jan 2017 14:09:07 -0800
Zac Medico  a écrit:

> On 01/10/2017 01:56 PM, Bernard Cafarelli wrote:
> > Hi folks,
> > 
> > gnustep-base/gnustep-make has a USE flag (libobjc2) masked globally in
> > base/package.use.mask, and unmasked on specific arches in
> > arch/{amd64,x86}/package.use.mask
> > 
> > To get a stabilization (#579232) bug finally moving on, I wanted to
> > leave this flag out, adding a corresponding line in
> > base/package.use.stable.mask
> > 
> > But repoman replied with a batch of dependency.bad errors...
> > Does package.use.mask (stable and ~arch) have a higher priority on
> > package.use.stable.mask (stable only)? Bug or intended behavior?
> > 
> > In the meantime, I will probably work around this by duplicating the
> > package.use.stable.mask entry in arches files
> >   
> 
> If I understand you correctly, then it's the intended behavior. If the
> flag is masked in both package.use.mask and package.use.stable.mask,
> then the package.use.stable.mask setting is irrelevant because both
> package.use.mask and package.use.stable.mask are considered when
> calculating use.mask settings for any given package.

Thanks for the confirmation! I fixed it at arch level:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ef57bcf1f6505b79d0cf696ba8f196df1d6f9c9c


-- 
Bernard Cafarelli (Voyageur)
Gentoo developer

Re: [gentoo-dev] package.use.mask / package.use.stable.mask priority

[Zac Medico]

On 01/10/2017 01:56 PM, Bernard Cafarelli wrote:
> Hi folks,
> 
> gnustep-base/gnustep-make has a USE flag (libobjc2) masked globally in
> base/package.use.mask, and unmasked on specific arches in
> arch/{amd64,x86}/package.use.mask
> 
> To get a stabilization (#579232) bug finally moving on, I wanted to
> leave this flag out, adding a corresponding line in
> base/package.use.stable.mask
> 
> But repoman replied with a batch of dependency.bad errors...
> Does package.use.mask (stable and ~arch) have a higher priority on
> package.use.stable.mask (stable only)? Bug or intended behavior?
> 
> In the meantime, I will probably work around this by duplicating the
> package.use.stable.mask entry in arches files
> 

If I understand you correctly, then it's the intended behavior. If the
flag is masked in both package.use.mask and package.use.stable.mask,
then the package.use.stable.mask setting is irrelevant because both
package.use.mask and package.use.stable.mask are considered when
calculating use.mask settings for any given package.
-- 
Thanks,
Zac

[gentoo-dev] [warning] the bug queue has 86 bugs

[Alex Alexander]

Our bug queue has 86 bugs!

If you have some spare time, please help assign/sort a few bugs.

To view the bug queue, click here: http://bit.ly/m8PQS5

Thanks!

[gentoo-dev] package.use.mask / package.use.stable.mask priority

[Bernard Cafarelli]

Hi folks,

gnustep-base/gnustep-make has a USE flag (libobjc2) masked globally in
base/package.use.mask, and unmasked on specific arches in
arch/{amd64,x86}/package.use.mask

To get a stabilization (#579232) bug finally moving on, I wanted to
leave this flag out, adding a corresponding line in
base/package.use.stable.mask

But repoman replied with a batch of dependency.bad errors...
Does package.use.mask (stable and ~arch) have a higher priority on
package.use.stable.mask (stable only)? Bug or intended behavior?

In the meantime, I will probably work around this by duplicating the
package.use.stable.mask entry in arches files

-- 
Bernard Cafarelli (Voyageur)
Gentoo developer

[gentoo-dev] Last-rites: www-plugins/pipelight

[NP-Hardass]

# NP-Hardass  (19 Jan 2017)
# Upstream has discontinued Pipelight and Firefox is in the process
# of removing NPAPI support (which Pipelight relies upon), making
# this obsolete.
# Masked for removal in 30 days.
www-plugins/pipelight


-- 
NP-Hardass



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] bzipped manpages

[Michał Górny]

On Tue, 10 Jan 2017 15:01:15 +0200
Mart Raudsepp  wrote:

> Ühel kenal päeval, T, 10.01.2017 kell 19:19, kirjutas Vadim A. Misbakh-
> Soloviov:
> > that will 
> > affect tons of users (which are happy with current "defaults")
> > because yours 
> > only own local problems (not having root access on the system)?  
> 
> Yes, the default should be changed for everyone.
> To PORTAGE_COMPRESS="xz".

Please do not encourage the use of 'wannabe-7zip' further. If you want
LZMA2, then lzip has less complexity, less overhead and slightly better
compression ratio than xz. It is also more reliable.

-- 
Best regards,
Michał Górny



pgp_KeX_aWQM3.pgp
Description: OpenPGP digital signature

[gentoo-dev] The changes about the stabilization process - part 2

[Agostino Sarubbo]

Hello All,

this message is an update of:
https://archives.gentoo.org/gentoo-dev/message/4b2ef0e9aa7588224b8ae799c5fe31fa 


What is changed in the meantime:

1) All operations on the bugzilla, about the sanity-check and so on are done 
with the account stable-...@gentoo.org
So, if you want to filter them, please use the 'X-Bugzilla-Who' flag.


2) We will move the "Atoms to stabilize" field to "Packages list" as pointed 
out by ulm here: 
https://archives.gentoo.org/gentoo-dev/message/0714886af9e9a233d8dc2b2d0a1f261e


3) We have two pages on the wiki that come from past contributions; they were 
not written recently and they need to be reviewed.
So consider the content a draft, but keep the links that will be permanent.

https://wiki.gentoo.org/wiki/Stable_request
https://wiki.gentoo.org/wiki/Package_testing

--
Agostino

Re: [gentoo-dev] bzipped manpages

[Michał Górny]

On Tue, 10 Jan 2017 12:54:21 +0100
Jan Stary  wrote:

> On Jan 09 09:30:11, ike...@gentoo.org wrote:
> > Hiya Jan,
> > 
> > The following snippet from Ingo is correct:
> >   
> > > So, you want to hear something constructive?  Your best option is to
> > > just decompress that stuff on your system.  (Gentoo is famous for
> > > its excessive configurability - maybe there is even an option?)  
> > 
> > We are both famous for our excessive configurability and there is even
> > an option already!  5:)  If you look in the manpage (once you've
> > decompress it somewhere, or online at [1]) for make.conf, you'll see the
> > entry for PORTAGE_COMPRESS, which you can set as follows:
> > 
> > PORTAGE_COMPRESS=""  
> 
> I am only a user on this system,
> and have no control over which packages are installed
> and have no write permissions in /usr/share/man/ or make.conf

If you are only a user, then why don't you contact your sysadmin
instead of trying to work around his choice at the distribution level?
After all, as you already know, he will need to rebuild everything
anyway.

> > As mentioned in [2,3,others].  You'll then need to reinstall all
> > packages.  If you manually decompress the files, then the uncompressed
> > manpages won't be registered with portage and won't get removed if the
> > owning package is uninstalled.  
> 
> Also, the uncompressed manpage will not get updated
> when the packages gets updated. I will have two copies,
> a stale *.1 and an up-to-date *.1.bz2.
> 
> These are workarounds. Let me get back to the original question:
> would you please consider having _uncompressed_ manpages as the default?
> 
> On this particular system, the bzipped /usr/share/man/ is 67M.
> The uncompressed man/ is 108M. That's 40M saved. Seriously?

~40% is a pretty good gain. However, if you really insist on comparing
this, few points to consider:

1. Since there are many small files involved, the results highly differ
depending on the filesystem used. On some filesystems (btrfs), it will
be very hard to even get any conclusive numbers.

2. The compression feature extends to all documentation,
including /usr/share/doc and /usr/share/info. So you should really
consider it all rather than limiting your view to manpages.

3. In some cases, the compression can also improve performance by
reducing I/O overhead. While it's debatable whether it will happen on
manpages (see filesystem problem above too), there is no real
performance loss to be considered either. After all, manpages are read
rather rarely in the lifetime of production system, so any effort in
decompressing them is a minor problem.

> There is an option to support; the packages need to be reinstalled
> or there are untracked files;

Are you arguing for removing the option altogether? Because as far as I
can see, the problems involved in changing the value are rather
an argument not to change it...

> the manpage formatter needs to call
> external unpackers. All this to save 40M. I honestly don't think
> it's worth it.

Calling external tools in a pipeline is a pretty normal solution
in the *nix world. It could be even considered safer than implementing
multiple disjoint features in a single tool where an issue in one
module could damage other parts of the program.

If you really do mind it, pretty much every compression format
supported by Gentoo provides a simple library you could use. There are
also a few libraries that provide support for multiple compression
formats.

To summarize, I'm afraid you don't have any arguments besides using
non-standard tool whose upstream refuses to support normal Gentoo
installations (is Gentoo really the only distribution using manpage
compression other than gzip?). You have multiple solutions available
that do not require Gentoo to suddenly change the defaults that work
for most of our users (and some of them appreciate them). Those
include:

1. using another man page tool,

2. writing a wrapper that would decompress manpages for your tool,

3. patching your tool to support bzip2 (I see that Fabian provided you
with a patch already),

4. talking to your sysadmin to update the system to meet your needs.

-- 
Best regards,
Michał Górny



pgpSZJFXUWuFE.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] bzipped manpages

[Mart Raudsepp]

Ühel kenal päeval, T, 10.01.2017 kell 14:39, kirjutas Ulrich Mueller:
> > > > > > On Tue, 10 Jan 2017, Mart Raudsepp wrote:
> > Yes, the default should be changed for everyone.
> > To PORTAGE_COMPRESS="xz".
> 
> Back in 2013, vapier had made extensive studies of compression tools
> for man pages and documentation, and the conclusion was that bzip2
> gives the best overall compression ratio for these files.
> https://bugs.gentoo.org/show_bug.cgi?id=169260

Nvm then, assuming it also gives the fastest streaming decompression
ratio, which I suspect is close enough with unxz.

My real point back of my mind was that these days it can be faster to
read a heavily compressed file from disk and decompress than to only
read a larger file from disk. And then we got SSDs, so that might be
different again.

Re: [gentoo-dev] bzipped manpages

[Ulrich Mueller]

> On Tue, 10 Jan 2017, Mart Raudsepp wrote:

> Yes, the default should be changed for everyone.
> To PORTAGE_COMPRESS="xz".

Back in 2013, vapier had made extensive studies of compression tools
for man pages and documentation, and the conclusion was that bzip2
gives the best overall compression ratio for these files.
https://bugs.gentoo.org/show_bug.cgi?id=169260

Ulrich


pgpDEGJDQs_fX.pgp
Description: PGP signature

Re: [gentoo-dev] bzipped manpages

[Fabian Groffen]

On 09-01-2017 09:08:22 +0100, Jan Stary wrote:
> The particular problem I am having is that http://mdocml.bsd.lv/ ,
> my manpage formatter of choice, does deliberately not support bzip
> (or any other outside decompressors for that matter).

Attached patch works for me.  XZ should be a similar exercise, a little
cleanup would be nice then though.

Fabian

-- 
Fabian Groffen
Gentoo on a different level
--- mdocml-1.13.4/read.c
+++ mdocml-1.13.4/read.c
@@ -38,6 +38,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "mandoc_aux.h"
 #include "mandoc.h"
@@ -62,6 +62,7 @@
 	enum mandoclevel  wlevel; /* ignore messages below this */
 	int		  options; /* parser options */
 	int		  gzip; /* current input file is gzipped */
+	int		  bzip; /* current input file is bzipp2ed */
 	int		  filenc; /* encoding of the current file */
 	int		  reparse_count; /* finite interp. stack */
 	int		  line; /* line number in the file */
@@ -610,6 +611,7 @@
 		struct buf *fb, int *with_mmap)
 {
 	gzFile		 gz;
+	BZFILE		*bz;
 	size_t		 off;
 	ssize_t		 ssz;
 
@@ -629,7 +629,7 @@
 	 * concerned that this is going to tank any machines.
 	 */
 
-	if (curp->gzip == 0 && S_ISREG(st.st_mode)) {
+	if (curp->gzip == 0 && curp->bzip == 0 && S_ISREG(st.st_mode)) {
 		if (st.st_size > 0x7fff) {
 			mandoc_msg(MANDOCERR_TOOLARGE, curp, 0, 0, NULL);
 			return 0;
@@ -639,11 +641,15 @@
 	}
 #endif
 
+	gz = NULL;
+	bz = NULL;
 	if (curp->gzip) {
 		if ((gz = gzdopen(fd, "rb")) == NULL)
 			err((int)MANDOCLEVEL_SYSERR, "%s", file);
-	} else
-		gz = NULL;
+	} else if (curp->bzip) {
+		if ((bz = BZ2_bzdopen(fd, "rb")) == NULL)
+			err((int)MANDOCLEVEL_SYSERR, "%s", file);
+	}
 
 	/*
 	 * If this isn't a regular file (like, say, stdin), then we must
@@ -663,9 +669,13 @@
 			}
 			resize_buf(fb, 65536);
 		}
-		ssz = curp->gzip ?
-		gzread(gz, fb->buf + (int)off, fb->sz - off) :
-		read(fd, fb->buf + (int)off, fb->sz - off);
+		if (curp->gzip) {
+			ssz = gzread(gz, fb->buf + (int)off, fb->sz - off);
+		} else if (curp->bzip) {
+			ssz = BZ2_bzread(bz, fb->buf + (int)off, fb->sz - off);
+		} else {
+		ssz = read(fd, fb->buf + (int)off, fb->sz - off);
+		}
 		if (ssz == 0) {
 			fb->sz = off;
 			return 1;
@@ -785,6 +795,7 @@
 	curp->file = file;
 	cp = strrchr(file, '.');
 	curp->gzip = (cp != NULL && ! strcmp(cp + 1, "gz"));
+	curp->bzip = (cp != NULL && ! strcmp(cp + 1, "bz2"));
 
 	/* First try to use the filename as it is. */
 
@@ -804,6 +815,13 @@
 			curp->gzip = 1;
 			return fd;
 		}
+		mandoc_asprintf(, "%s.bz2", file);
+		fd = open(cp, O_RDONLY);
+		free(cp);
+		if (fd != -1) {
+			curp->bzip = 1;
+			return fd;
+		}
 	}
 
 	/* Neither worked, give up. */
--- mdocml-1.13.4/configure
+++ mdocml-1.13.4/configure
@@ -255,7 +255,7 @@
 fi
 
 # --- LDADD ---
-LDADD="${LDADD} ${LD_SQLITE3} ${LD_OHASH} -lz"
+LDADD="${LDADD} ${LD_SQLITE3} ${LD_OHASH} -lz -lbz2"
 echo "LDADD=\"${LDADD}\"" 1>&2
 echo "LDADD=\"${LDADD}\"" 1>&3
 echo 1>&3


signature.asc
Description: Digital signature

Re: [gentoo-dev] bzipped manpages

[Mart Raudsepp]

Ühel kenal päeval, T, 10.01.2017 kell 19:19, kirjutas Vadim A. Misbakh-
Soloviov:
> that will 
> affect tons of users (which are happy with current "defaults")
> because yours 
> only own local problems (not having root access on the system)?

Yes, the default should be changed for everyone.
To PORTAGE_COMPRESS="xz".

Note that it doesn't only affect man pages, but also tons of
/usr/share/doc/$PF/* files.

Re: [gentoo-dev] bzipped manpages

[Jan Stary]

On Jan 10 19:19:03, gen...@mva.name wrote:
> В письме от вторник, 10 января 2017 г. 13:08:14 +07 пользователь Jan Stary 
> написал:
> > On Jan 10 19:04:47, gen...@mva.name wrote:
> > > > There is an option to support; the packages need to be reinstalled
> > > > or there are untracked files; the manpage formatter needs to call
> > > > external unpackers. All this to save 40M. I honestly don't think
> > > > it's worth it.
> > > 
> > > Why do you care about calling external unpacker,
> > > but do not care about saving 40MB?
> > 
> > Because not having to call an external unpacker
> > allows for the manpage formatter to be simple;
> > whereas saving 40M of space is of no concern.
> 
> You arguing that 40MB is nothing on modern systems (which, by the way is not 
> exactly true, talking about embedded ones).

Can you gove an example of an embedded system with manpages?

> So, I guess, it means, that you have modern powerfull hardware, which is 
> pretty fine with some overheads.

If having an extra 40MB is "modern, powerfull hardware", then yes.

> Then why do you need "simple" manpage formatter?

Why do I want software to be simple?

> And actually, why calling external unpacker is so complicated? Almost any 
> programming language I know, has functions identical to C's system()...

It's not that complicated; it's unneeded, it's another dependency, etc.

> Do you fully understand, that you asking to change "defaults", that will 
> affect tons of users (which are happy with current "defaults") because yours 
> only own local problems (not having root access on the system)?

This has nothing to do with not having root - that only makes it
unable for me to use the _workarounds_.

What would be the effect of having uncompressed manpages as the default?
(Besides having them renderred by any manpage formatter,
and wasting 40MB of space, obviously)?

Jan

Re: [gentoo-dev] bzipped manpages

[Vadim A. Misbakh-Soloviov]

В письме от вторник, 10 января 2017 г. 13:08:14 +07 пользователь Jan Stary 
написал:
> On Jan 10 19:04:47, gen...@mva.name wrote:
> > > There is an option to support; the packages need to be reinstalled
> > > or there are untracked files; the manpage formatter needs to call
> > > external unpackers. All this to save 40M. I honestly don't think
> > > it's worth it.
> > 
> > Why do you care about calling external unpacker,
> > but do not care about saving 40MB?
> 
> Because not having to call an external unpacker
> allows for the manpage formatter to be simple;
> whereas saving 40M of space is of no concern.

You arguing that 40MB is nothing on modern systems (which, by the way is not 
exactly true, talking about embedded ones).
So, I guess, it means, that you have modern powerfull hardware, which is 
pretty fine with some overheads.

Then why do you need "simple" manpage formatter? Why don't use all of that 
complicated ones (man-db, vim's Man.vim, whatever) instead?

P.S.:

And actually, why calling external unpacker is so complicated? Almost any 
programming language I know, has functions identical to C's system()...

P.P.S:

Do you fully understand, that you asking to change "defaults", that will 
affect tons of users (which are happy with current "defaults") because yours 
only own local problems (not having root access on the system)?

Re: [gentoo-dev] bzipped manpages

[Jan Stary]

On Jan 10 19:04:47, gen...@mva.name wrote:
> > There is an option to support; the packages need to be reinstalled
> > or there are untracked files; the manpage formatter needs to call
> > external unpackers. All this to save 40M. I honestly don't think
> > it's worth it.
> 
> Why do you care about calling external unpacker,
> but do not care about saving 40MB?

Because not having to call an external unpacker
allows for the manpage formatter to be simple;
whereas saving 40M of space is of no concern.

Re: [gentoo-dev] bzipped manpages

[Jan Stary]

> > This is Gentoo 2.2 (4.4.6-gentoo x86_64).
> 
> That doesn't actually tell any Gentoo user anything about your system
> except a very specific few bits of data which do not relate at all to
> the rest of the subject matter of your e-mail.

I am not really familiar eith this system - what would be
the right piece of information that does relate tot this?

> [1] echo 'PORTAGE_COMPRESS=""' >> /etc/portage/make.conf

-ksh: /etc/portage/make.conf: cannot create [Permission denied]

Jan

Re: [gentoo-dev] bzipped manpages

[Vadim A. Misbakh-Soloviov]

> There is an option to support; the packages need to be reinstalled
> or there are untracked files; the manpage formatter needs to call
> external unpackers. All this to save 40M. I honestly don't think
> it's worth it.

Why do you care about calling external unpacker, but do not care about saving 
40MB?

Double standards?

Re: [gentoo-dev] bzipped manpages

[Jan Stary]

On Jan 10 12:54:21, h...@stare.cz wrote:
> Also, the uncompressed manpage will not get updated
> when the packages gets updated. I will have two copies,
> a stale *.1 and an up-to-date *.1.bz2.

And things like /usr/share/man/man1/sx.1.bz2
will not get unbzipped, because it's a symlink, now broken.
So a special care needs to be taken of the _names_ of these.

Jan

Re: [gentoo-dev] bzipped manpages

[Jan Stary]

On Jan 09 09:30:11, ike...@gentoo.org wrote:
> Hiya Jan,
> 
> The following snippet from Ingo is correct:
> 
> > So, you want to hear something constructive?  Your best option is to
> > just decompress that stuff on your system.  (Gentoo is famous for
> > its excessive configurability - maybe there is even an option?)
> 
> We are both famous for our excessive configurability and there is even
> an option already!  5:)  If you look in the manpage (once you've
> decompress it somewhere, or online at [1]) for make.conf, you'll see the
> entry for PORTAGE_COMPRESS, which you can set as follows:
> 
> PORTAGE_COMPRESS=""

I am only a user on this system,
and have no control over which packages are installed
and have no write permissions in /usr/share/man/ or make.conf

> As mentioned in [2,3,others].  You'll then need to reinstall all
> packages.  If you manually decompress the files, then the uncompressed
> manpages won't be registered with portage and won't get removed if the
> owning package is uninstalled.

Also, the uncompressed manpage will not get updated
when the packages gets updated. I will have two copies,
a stale *.1 and an up-to-date *.1.bz2.

These are workarounds. Let me get back to the original question:
would you please consider having _uncompressed_ manpages as the default?

On this particular system, the bzipped /usr/share/man/ is 67M.
The uncompressed man/ is 108M. That's 40M saved. Seriously?

There is an option to support; the packages need to be reinstalled
or there are untracked files; the manpage formatter needs to call
external unpackers. All this to save 40M. I honestly don't think
it's worth it.

Jan